Jul 28 14:20:44.133129 ip-10-0-142-230 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 14:20:44.133142 ip-10-0-142-230 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 14:20:44.133152 ip-10-0-142-230 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 14:20:44.133464 ip-10-0-142-230 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 14:20:54.157061 ip-10-0-142-230 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 14:20:54.157077 ip-10-0-142-230 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2b8fca5f59214c6ba7f4d85c6ffbf5d1 --
Jul 28 14:22:42.201585 ip-10-0-142-230 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 14:22:42.595916 ip-10-0-142-230 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:42.595916 ip-10-0-142-230 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 14:22:42.595916 ip-10-0-142-230 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:42.595916 ip-10-0-142-230 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 14:22:42.595916 ip-10-0-142-230 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:42.597028 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.596905    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 14:22:42.600482 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600463    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.600482 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600482    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600486    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600490    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600494    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600497    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600500    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600502    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600505    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600508    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600511    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600513    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600518    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600522    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600525    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600527    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600530    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600548    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600552    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600556    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.600559 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600560    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600566    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600569    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600572    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600574    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600577    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600580    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600582    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600585    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600587    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600590    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600593    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600596    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600598    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600601    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600603    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600605    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600608    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600611    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600613    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.601046 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600616    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600618    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600620    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600623    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600626    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600629    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600634    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600638    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600641    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600644    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600646    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600658    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600661    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600664    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600667    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600670    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600672    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600675    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600678    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.601549 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600680    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600684    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600686    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600689    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600691    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600694    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600697    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600699    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600702    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600704    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600708    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600712    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600715    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600718    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600720    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600723    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600725    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600728    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600731    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600734    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.602019 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600736    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600739    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600742    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600744    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600747    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600749    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600753    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600756    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600758    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600761    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600763    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600766    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600768    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600771    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600773    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600776    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600778    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600781    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600783    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.600787    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.602507 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602161    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602169    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602172    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602175    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602179    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602184    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602187    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602190    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602194    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602198    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602201    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602204    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602206    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602209    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602213    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602216    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602219    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602222    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602224    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.603003 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602227    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602230    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602232    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602235    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602238    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602240    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602243    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602246    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602249    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602252    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602254    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602257    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602260    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602263    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602265    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602268    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602271    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602273    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602276    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602279    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.603488 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602282    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602285    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602287    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602290    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602292    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602295    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602297    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602300    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602303    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602306    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602308    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602311    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602313    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602316    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602318    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602321    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602324    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602326    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602332    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.604004 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602334    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602337    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602340    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602342    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602344    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602347    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602349    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602352    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602354    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602357    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602360    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602362    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602365    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602368    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602370    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602373    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602375    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602378    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602380    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602384    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.604512 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602387    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602389    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602392    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602396    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602399    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602402    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602405    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602408    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602410    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602413    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602415    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602418    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602420    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602423    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602425    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602428    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602430    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602433    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602435    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602437    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.605011 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602832    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602837    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602840    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602844    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602847    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602850    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602853    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602856    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602858    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602861    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602863    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602866    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602869    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602872    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602874    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602877    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602880    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602882    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602885    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602887    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.605497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602890    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602892    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602895    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602898    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602900    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602903    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602905    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602908    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602910    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602913    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602915    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602918    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602920    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602923    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602925    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602928    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602931    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602934    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602937    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602940    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.606027 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602942    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602945    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602947    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602951    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602953    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602956    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602958    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602961    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602963    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602965    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602968    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602971    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602973    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602976    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602978    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602981    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602983    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602985    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602988    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.606515 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602991    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602995    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.602999    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603002    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603005    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603008    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603011    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603014    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603018    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603022    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603026    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603030    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603033    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603036    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603038    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603041    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603044    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603046    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.607021 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603048    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603053    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603056    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603059    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603061    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603064    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603066    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603069    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603071    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603074    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603076    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603079    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603081    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603084    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603086    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603089    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603091    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603094    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603096    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603098    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.607484 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603101    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603103    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603182    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603190    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603196    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603200    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603206    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603209    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603213    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603221    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603225    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603228    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603231    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603234    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603238    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603241    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603244    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603247    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603250    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603253    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603258    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603261    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603264    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603267    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 14:22:42.608100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603271    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603275    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603278    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603281    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603285    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603288    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603291    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603295    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603298    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603301    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603306    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603309    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603312    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603315    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603318    2577 flags.go:64] FLAG: --enable-server="true"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603322    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603326    2577 flags.go:64] FLAG: --event-burst="100"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603330    2577 flags.go:64] FLAG: --event-qps="50"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603333    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603336    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603339    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603343    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603346    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603349    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603351    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 28 14:22:42.608707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603354    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603357    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603360    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603363    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603366    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603369    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603372    2577 flags.go:64] FLAG: --feature-gates=""
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603376    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603379    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603381    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603385    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603388    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603392    2577 flags.go:64] FLAG: --help="false"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603395    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603398    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603401    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603404    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603408    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603412    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603415    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603418    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603420    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603423    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 14:22:42.609307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603427    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603430    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603433    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603436    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603439    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603442    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603445    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603448    2577 flags.go:64] FLAG: --lock-file=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603450    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603453    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603456    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603462    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603465    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603468    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603470    2577 flags.go:64] FLAG: --logging-format="text"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603473    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603477    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603479    2577 flags.go:64] FLAG: --manifest-url=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603482    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603486    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603490    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603494    2577 flags.go:64] FLAG: --max-pods="110"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603497    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603500    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603503    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 14:22:42.609988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603506    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603510    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603513    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603516    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603523    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603526    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603530    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603546    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603549    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603555    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603558    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603561    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603564    2577 flags.go:64] FLAG: --port="10250"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603567    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603570    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0820f3c84e730789f"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603573    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603584    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603587    2577 flags.go:64] FLAG: --register-node="true"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603590    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603594    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603597    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603599    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603602    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603607    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603610    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 14:22:42.610611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603613    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603616    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603619    2577 flags.go:64] FLAG: --runonce="false"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603622    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603626    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603629    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603632    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603635    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603638    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603643    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603646    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603650    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603653    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603656    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603659    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603662    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603667    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603670    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603676    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603678    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603681    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603686    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603689    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603692    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603695    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 14:22:42.611267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603698    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603701    2577 flags.go:64] FLAG: --v="2"
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603705    2577 flags.go:64] FLAG: --version="false"
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603709    2577 flags.go:64] FLAG: --vmodule=""
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603713    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.603720    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603853    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603857    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603860    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603863    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603865    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603868    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603871    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603874    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603877    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603879    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603883    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603886    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603888    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603891    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603894    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.611891 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603897    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603899    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603902    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603911    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603929    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603933    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603936    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603939    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603942    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603945    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603948    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603950    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603953    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603956    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603958    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603961    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603966    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603969    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603971    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603974    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.612413 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603977    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603979    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603982    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603984    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603987    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603989    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603993    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603996    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.603999    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604001    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604004    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604007    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604010    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604012    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604015    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604018    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604027    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604029    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604032    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604034    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.612910 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604037    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604039    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604042    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604045    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604047    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604050    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604052    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604054    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604058    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604060    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604063    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604067    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604071    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604073    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604076    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604078    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604081    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604083    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604086    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.613392 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604089    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604092    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604094    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604098    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604102    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604104    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604107    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604109    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604112    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604114    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604125    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604128    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604130    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604133    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604135    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604138    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604142    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604145    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604148    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.613871 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604151    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604154    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604158    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604161    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604163    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.604166    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.604680    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.611631    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.611647    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.611668    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.611673    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611749    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611755    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611758    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611761    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.614336 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611764    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611766    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611769    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611772    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611775    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611778    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611780    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611783    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611792    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611795    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611797    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611800    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611802    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611805    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611807    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611810    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611813    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611817    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611821    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611824    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.614843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611827    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611830    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611833    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611835    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611838    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611842    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611844    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611847    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611850    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611852    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611855    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611857    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611860    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611864    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611870    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611873    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611876    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611879    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611881    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.615346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611884    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611886    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611895    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611898    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611900    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611903    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611906    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611908    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611911    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611914    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611916    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611919    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611921    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611924    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611926    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611929    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611932    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611934    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611938    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611940    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.615843 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611943    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611946    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611948    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611951    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611955    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611958    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611961    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611964    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611967    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611970    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611973    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611976    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611978    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611980    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611983    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611991    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611993    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611996    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.611998    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.616350 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612001    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612003    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612006    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612008    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612011    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612020    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612022    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612025    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612027    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612030    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612032    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612036    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612038    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612041    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612044    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612046    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.616834 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612050    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.612055    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612215    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612221    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612224    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612226    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612229    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612233    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612237    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612240    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612243    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612246    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612248    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612256    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612259    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:42.617218 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612262    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612264    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612267    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612269    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612272    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612274    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612277    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612279    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612282    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612284    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612288    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612291    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612295    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612298    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612301    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612303    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612306    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612309    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612311    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:42.617598 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612314    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612316    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612319    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612321    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612324    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612326    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612329    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612332    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612334    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612337    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612339    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612342    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612350    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612352    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612355    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612358    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612360    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612363    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612366    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612369    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:42.618059 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612371    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612374    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612377    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612379    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612382    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612384    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612387    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612390    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612392    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612395    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612397    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612400    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612402    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612404    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612407    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612409    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612412    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612414    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612417    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612420    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:42.618544 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612423    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612426    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612428    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612431    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612433    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612441    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612444    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612446    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612449    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612452    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612455    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612457    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612460    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612462    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612465    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612468    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612472    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612475    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612478    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:42.619288 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612480    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612483    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612485    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612488    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612490    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612492    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612495    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:42.612497    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.612502    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.612658    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.614687    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.616123    2577 server.go:1019] "Starting client certificate rotation"
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.616222    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 14:22:42.620086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.616255    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 14:22:42.621031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.621009    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 14:22:42.625745 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.625726    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 14:22:42.639403 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.639377    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 28 14:22:42.641291 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.641273    2577 log.go:25] "Validated CRI v1 image API"
Jul 28 14:22:42.642126 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.642105    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 14:22:42.644420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.644400    2577 fs.go:135] Filesystem UUIDs: map[5f368d58-e55f-43fb-be40-d7aa7586ac68:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8fccf268-1ccb-45b3-9c25-8d5b3688164b:/dev/nvme0n1p4]
Jul 28 14:22:42.644482 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.644419    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 14:22:42.646016 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.645998    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 14:22:42.650165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.650047    2577 manager.go:216] Machine: {Timestamp:2026-07-28 14:22:42.648786612 +0000 UTC m=+0.334864716 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114383 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26f216cb79bff270cf78c5f0fb3ac5 SystemUUID:ec26f216-cb79-bff2-70cf-78c5f0fb3ac5 BootID:2b8fca5f-5921-4c6b-a7f4-d85c6ffbf5d1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:92:a6:7a:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:92:a6:7a:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:c5:3b:32:79:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 14:22:42.650165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.650159    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 14:22:42.650341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.650245    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 14:22:42.653259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.653225    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 14:22:42.653432 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.653261    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 14:22:42.654509 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.654493    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 14:22:42.654564 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.654512    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 14:22:42.654564 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.654520    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 14:22:42.655138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.655127    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 14:22:42.655895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.655884    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 14:22:42.657526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657514    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 14:22:42.657661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657651    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 14:22:42.657701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657682    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 14:22:42.657701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657691    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 14:22:42.657778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657712    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 14:22:42.657778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657725    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 28 14:22:42.657778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.657747    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 14:22:42.658321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.658304    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 14:22:42.658321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.658325    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 14:22:42.660481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.660464    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 14:22:42.662293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.662278    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 14:22:42.662340 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.662312    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 14:22:42.663817 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663805    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 14:22:42.663852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663823    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 14:22:42.663852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663830    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 14:22:42.663852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663835    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 14:22:42.663852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663844    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 14:22:42.663852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663849    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663857    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663863    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663869    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663874    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663888    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 14:22:42.663991 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.663900    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 14:22:42.664598 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.664589    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 14:22:42.664598 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.664598    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 14:22:42.668449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.668165    2577 server.go:1268] "Started kubelet"
Jul 28 14:22:42.668449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.668336    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 14:22:42.669202 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.669115    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 14:22:42.669302 ip-10-0-142-230 systemd[1]: Started Kubernetes Kubelet.
Jul 28 14:22:42.669493 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.669318    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 14:22:42.670321 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.670059    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 14:22:42.670395 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.670324    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 14:22:42.670656 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.670060    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 14:22:42.671100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.671061    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 14:22:42.673989 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.673972    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 14:22:42.674473 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.674454    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 14:22:42.676836 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.675797    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-230.ec2.internal.18c67a1bed978612  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-230.ec2.internal,UID:ip-10-0-142-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-230.ec2.internal,},FirstTimestamp:2026-07-28 14:22:42.667816466 +0000 UTC m=+0.353894570,LastTimestamp:2026-07-28 14:22:42.667816466 +0000 UTC m=+0.353894570,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-230.ec2.internal,}"
Jul 28 14:22:42.679500 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.679481    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 14:22:42.679500 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.679491    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 14:22:42.679650 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.679591    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:42.680088 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680075    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 14:22:42.680263 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680243    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 14:22:42.680263 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680259    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 14:22:42.680461 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680437    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 14:22:42.680548 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680455    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 14:22:42.680716 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680696    2577 factory.go:55] Registering systemd factory
Jul 28 14:22:42.680716 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680717    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 28 14:22:42.680959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680942    2577 factory.go:153] Registering CRI-O factory
Jul 28 14:22:42.681035 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.680963    2577 factory.go:223] Registration of the crio container factory successfully
Jul 28 14:22:42.681035 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.681024    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 14:22:42.681124 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.681054    2577 factory.go:103] Registering Raw factory
Jul 28 14:22:42.681124 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.681070    2577 manager.go:1196] Started watching for new ooms in manager
Jul 28 14:22:42.681560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.681530    2577 manager.go:349] Starting recovery of all containers
Jul 28 14:22:42.687330 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.687288    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 14:22:42.689147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.689118    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csq75"
Jul 28 14:22:42.690926 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.690904    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 14:22:42.691303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.691163    2577 manager.go:354] Recovery completed
Jul 28 14:22:42.691586 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.691446    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 14:22:42.696529 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.696506    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csq75"
Jul 28 14:22:42.697159 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.697143    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.700416 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.700400    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.700485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.700427    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.700485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.700438    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.701038 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701021    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 14:22:42.701038 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701035    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 14:22:42.701161 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701054    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 14:22:42.701161 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701028    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.701944 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701927    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.702024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701958    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.702024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.701973    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.702656 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.702588    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-230.ec2.internal.18c67a1bef88ed3f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-230.ec2.internal,UID:ip-10-0-142-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-230.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-230.ec2.internal,},FirstTimestamp:2026-07-28 14:22:42.700414271 +0000 UTC m=+0.386492377,LastTimestamp:2026-07-28 14:22:42.700414271 +0000 UTC m=+0.386492377,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-230.ec2.internal,}"
Jul 28 14:22:42.703684 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.703668    2577 policy_none.go:49] "None policy: Start"
Jul 28 14:22:42.703684 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.703686    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 14:22:42.703787 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.703697    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 14:22:42.704724 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.704713    2577 policy_none.go:47] "Start"
Jul 28 14:22:42.748463 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.748273    2577 manager.go:337] "Starting Device Plugin manager"
Jul 28 14:22:42.748607 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.748492    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 14:22:42.748607 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.748509    2577 server.go:87] "Starting device plugin registration server"
Jul 28 14:22:42.748817 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.748805    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 14:22:42.748870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.748819    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 14:22:42.748952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.748928    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 14:22:42.749053 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.749039    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 14:22:42.749053 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.749053    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 14:22:42.749639 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.749596    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 14:22:42.749750 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.749680    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:42.779168 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.779137    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 14:22:42.780106 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.779193    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 14:22:42.780106 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.779216    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 14:22:42.780106 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.779252    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 14:22:42.782370 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.782349    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:42.851983 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.851881    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.853024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.853002    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.853105 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.853041    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.853105 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.853070    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.853105 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.853100    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.862946 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.862918    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.863022 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.862952    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-230.ec2.internal\": node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:42.876836 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.876809    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:42.879850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.879833    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"]
Jul 28 14:22:42.879907 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.879898    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.880856 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.880837    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.880961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.880861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.880961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.880884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.880961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.880868    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.880961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.880921    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.882135 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882122    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.882281 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.882348 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882302    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.882897 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882879    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.883002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882910    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.883002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882928    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.883002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.882981    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.883119 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.883006    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.883119 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.883024    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.884083 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.884065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.884172 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.884099    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:42.884838 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.884824    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:42.884934 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.884852    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:42.884934 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.884867    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:42.911564 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.911531    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-230.ec2.internal\" not found" node="ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.915523 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.915503    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-230.ec2.internal\" not found" node="ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.977178 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:42.977143    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:42.981825 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.981801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.981899 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.981834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.981899 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.981856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/206bac433f2fce4d8e4d8f2249cfb20b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"206bac433f2fce4d8e4d8f2249cfb20b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.981973 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.981915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:42.981973 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:42.981922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c227a17b6b32368aeb98bed0f11f5ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal\" (UID: \"1c227a17b6b32368aeb98bed0f11f5ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:43.077964 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.077927    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.082650 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.082626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/206bac433f2fce4d8e4d8f2249cfb20b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"206bac433f2fce4d8e4d8f2249cfb20b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:43.082707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.082694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/206bac433f2fce4d8e4d8f2249cfb20b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-230.ec2.internal\" (UID: \"206bac433f2fce4d8e4d8f2249cfb20b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:43.179297 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.179212    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.216463 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.216428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:43.218724 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.218704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:43.279384 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.279349    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.386898 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.386858    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.487473 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.487381    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.588292 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.588258    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.611155 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.611122    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:43.615115 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.615094    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 14:22:43.615251 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.615218    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 14:22:43.615290 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.615266    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 14:22:43.688430 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.688397    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.701017 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.700963    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 14:17:43 +0000 UTC" deadline="2028-04-16 14:01:47.678027866 +0000 UTC"
Jul 28 14:22:43.701017 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.701011    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15071h39m3.977020376s"
Jul 28 14:22:43.788758 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.788678    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.801552 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:43.801505    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206bac433f2fce4d8e4d8f2249cfb20b.slice/crio-629dbe133858d2e8fe4c548a5bb05b81434cc6346949165497638caa02720b4e WatchSource:0}: Error finding container 629dbe133858d2e8fe4c548a5bb05b81434cc6346949165497638caa02720b4e: Status 404 returned error can't find the container with id 629dbe133858d2e8fe4c548a5bb05b81434cc6346949165497638caa02720b4e
Jul 28 14:22:43.809435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.809414    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:22:43.815465 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:43.815439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c227a17b6b32368aeb98bed0f11f5ed.slice/crio-5f8cb33146dc8e320f1ec408243ccaf1680e48819e46c8d10f6b12208e6e843a WatchSource:0}: Error finding container 5f8cb33146dc8e320f1ec408243ccaf1680e48819e46c8d10f6b12208e6e843a: Status 404 returned error can't find the container with id 5f8cb33146dc8e320f1ec408243ccaf1680e48819e46c8d10f6b12208e6e843a
Jul 28 14:22:43.865790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:43.865758    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:43.889776 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.889746    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:43.990568 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:43.990523    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:44.091458 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.091359    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:44.192610 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.192575    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-230.ec2.internal\" not found"
Jul 28 14:22:44.192897 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.192878    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:44.279898 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.279865    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:44.294672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.294646    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 14:22:44.295769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.295749    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal"
Jul 28 14:22:44.306372 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.306351    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 14:22:44.659986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.659948    2577 apiserver.go:52] "Watching apiserver"
Jul 28 14:22:44.667048 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.667024    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 14:22:44.668580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.668546    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-jntvr","openshift-ovn-kubernetes/ovnkube-node-9dgs4","kube-system/global-pull-secret-syncer-4fzdm","openshift-image-registry/node-ca-ckjp9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-5fktm","openshift-multus/multus-zq4qp","openshift-multus/network-metrics-daemon-sqcgv","openshift-network-diagnostics/network-check-target-x2n29","openshift-network-operator/iptables-alerter-f5rfw","kube-system/konnectivity-agent-vgnhf","kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr","openshift-cluster-node-tuning-operator/tuned-xrk55"]
Jul 28 14:22:44.670210 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.670188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.671545 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.671498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.673143 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673118    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.673250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673176    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 14:22:44.673250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673185    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.673250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673241    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 14:22:44.673419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zttf7\""
Jul 28 14:22:44.673778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.673756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.673957 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.673838    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:44.674223 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.674191    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rjqjh\""
Jul 28 14:22:44.674335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.674293    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 14:22:44.674382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.674352    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 14:22:44.675425 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.675404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.676901 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.676881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.677520 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.677463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cqncs\""
Jul 28 14:22:44.677939 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.677920    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.678162 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.678146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.678227 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.678199    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 14:22:44.678983 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.678963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.679381 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.679361    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.679857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.679724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tcrx4\""
Jul 28 14:22:44.682465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.681285    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 14:22:44.682465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.681347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.683213 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.682859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:44.683213 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.682962    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:44.683213 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.683023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 14:22:44.683410 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.683288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-chbrv\""
Jul 28 14:22:44.683410 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.683333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 14:22:44.691573 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.691551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.693196 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.693173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.694190 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.694168    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.694277 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.694207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.694537 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.694515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 14:22:44.694708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.694688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-conf\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l5w6j\""
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-systemd\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-run\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-os-release\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqx8\" (UniqueName: \"kubernetes.io/projected/a9641f09-cad8-4bde-9109-1a7da203da71-kube-api-access-hmqx8\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-modprobe-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-host\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-system-cni-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-etc-kubernetes\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-os-release\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.696163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-socket-dir-parent\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.695954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-multus\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-hostroot\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-conf-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-daemon-config\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktbd\" (UniqueName: \"kubernetes.io/projected/cc6329f8-e3de-4d56-8353-c1be192f862c-kube-api-access-bktbd\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbtv\" (UniqueName: \"kubernetes.io/projected/c2953d14-8846-4a93-9f00-03b4e7eca783-kube-api-access-wtbtv\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zpk\" (UniqueName: \"kubernetes.io/projected/99b6f0b0-9666-4042-ab4a-1887720c36ef-kube-api-access-w6zpk\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-cnibin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-konnectivity-ca\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-kubelet-config\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-tmp\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-cni-binary-copy\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697089 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-netns\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-agent-certs\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-dbus\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9641f09-cad8-4bde-9109-1a7da203da71-serviceca\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-var-lib-kubelet\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-bin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-kubelet\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-kubernetes\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-lib-modules\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-system-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysconfig\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-sys\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-tuned\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-cnibin\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696817    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-multus-certs\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.697876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9641f09-cad8-4bde-9109-1a7da203da71-host\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.696971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697118    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gvrj\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697158    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697656    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9c2vm\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.697664    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697674    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697730    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.698658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.697817    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 14:22:44.699073 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.698722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 14:22:44.699073 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.698781    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpwgk\""
Jul 28 14:22:44.699073 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.698838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 14:22:44.723734 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.723706    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s9x6p"
Jul 28 14:22:44.733267 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.733243    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s9x6p"
Jul 28 14:22:44.781557 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.781525    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 14:22:44.784850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.784804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"1c227a17b6b32368aeb98bed0f11f5ed","Type":"ContainerStarted","Data":"5f8cb33146dc8e320f1ec408243ccaf1680e48819e46c8d10f6b12208e6e843a"}
Jul 28 14:22:44.786001 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.785977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" event={"ID":"206bac433f2fce4d8e4d8f2249cfb20b","Type":"ContainerStarted","Data":"629dbe133858d2e8fe4c548a5bb05b81434cc6346949165497638caa02720b4e"}
Jul 28 14:22:44.797190 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9641f09-cad8-4bde-9109-1a7da203da71-serviceca\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.797320 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-var-lib-kubelet\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.797320 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-bin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.797320 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kmf\" (UniqueName: \"kubernetes.io/projected/8cdb37a7-d306-4fd0-94ff-dc390a119957-kube-api-access-n4kmf\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-var-lib-kubelet\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-bin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-socket-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.797455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-netd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-cnibin\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0c388cc-d3a8-4cee-aea7-8eadddae330e-tmp-dir\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-cnibin\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-registration-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-slash\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-config\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-script-lib\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-conf\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.797821    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-os-release\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.797871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9641f09-cad8-4bde-9109-1a7da203da71-serviceca\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.797886    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.297863945 +0000 UTC m=+2.983942050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-netns\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.797965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqx8\" (UniqueName: \"kubernetes.io/projected/a9641f09-cad8-4bde-9109-1a7da203da71-kube-api-access-hmqx8\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-modprobe-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-system-cni-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-conf\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-os-release\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-etc-kubernetes\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-agent-certs\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-etc-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-os-release\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-socket-dir-parent\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-multus\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-hostroot\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.798449 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-conf-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktbd\" (UniqueName: \"kubernetes.io/projected/cc6329f8-e3de-4d56-8353-c1be192f862c-kube-api-access-bktbd\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-node-log\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbtv\" (UniqueName: \"kubernetes.io/projected/c2953d14-8846-4a93-9f00-03b4e7eca783-kube-api-access-wtbtv\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-cnibin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-systemd-units\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-var-lib-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-kubelet-config\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-tmp\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-log-socket\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798681    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-os-release\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-socket-dir-parent\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-cni-multus\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-modprobe-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.799319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-hostroot\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-conf-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99b6f0b0-9666-4042-ab4a-1887720c36ef-system-cni-dir\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-cnibin\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-kubelet-config\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.798687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrfw\" (UniqueName: \"kubernetes.io/projected/0c826ad3-a5c4-47ee-9412-c87ebb211883-kube-api-access-jgrfw\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-dbus\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-etc-kubernetes\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-kubelet\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-var-lib-kubelet\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-sys-fs\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-dbus\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-kubernetes\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-lib-modules\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-kubernetes\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-system-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.800191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfgm\" (UniqueName: \"kubernetes.io/projected/b31b90ef-b78e-40a5-89c7-b36dabbc993d-kube-api-access-2lfgm\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-system-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysconfig\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-sys\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-lib-modules\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-tuned\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-cni-dir\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-multus-certs\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysconfig\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnqrp\" (UniqueName: \"kubernetes.io/projected/d0c388cc-d3a8-4cee-aea7-8eadddae330e-kube-api-access-qnqrp\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-multus-certs\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-device-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9641f09-cad8-4bde-9109-1a7da203da71-host\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-systemd\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9641f09-cad8-4bde-9109-1a7da203da71-host\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-run\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-run\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.801090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-konnectivity-ca\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-sysctl-d\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-host\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cdb37a7-d306-4fd0-94ff-dc390a119957-iptables-alerter-script\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.799978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-systemd\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-daemon-config\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdb37a7-d306-4fd0-94ff-dc390a119957-host-slash\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0c388cc-d3a8-4cee-aea7-8eadddae330e-hosts-file\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-ovn\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-bin\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zpk\" (UniqueName: \"kubernetes.io/projected/99b6f0b0-9666-4042-ab4a-1887720c36ef-kube-api-access-w6zpk\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpp5x\" (UniqueName: \"kubernetes.io/projected/57a7b3d9-e8ca-40f4-8960-0d522d492162-kube-api-access-cpp5x\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-kubelet\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-systemd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-cni-binary-copy\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-host\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-netns\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-env-overrides\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-netns\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-konnectivity-ca\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.800701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc6329f8-e3de-4d56-8353-c1be192f862c-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.801041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99b6f0b0-9666-4042-ab4a-1887720c36ef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.801127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-multus-daemon-config\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.801314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc6329f8-e3de-4d56-8353-c1be192f862c-cni-binary-copy\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.801373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2953d14-8846-4a93-9f00-03b4e7eca783-sys\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.802468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-tmp\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.802768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.802515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2953d14-8846-4a93-9f00-03b4e7eca783-etc-tuned\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.803337 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.802862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0cdcf6e-c202-43bb-8653-31f0c22b0e5a-agent-certs\") pod \"konnectivity-agent-vgnhf\" (UID: \"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a\") " pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:44.810833 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.810807    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:44.810948 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.810841    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:44.810948 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.810856    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:44.810948 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.810912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.310892474 +0000 UTC m=+2.996970571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:44.813968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.813939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zpk\" (UniqueName: \"kubernetes.io/projected/99b6f0b0-9666-4042-ab4a-1887720c36ef-kube-api-access-w6zpk\") pod \"multus-additional-cni-plugins-5fktm\" (UID: \"99b6f0b0-9666-4042-ab4a-1887720c36ef\") " pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:44.814100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.813969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqx8\" (UniqueName: \"kubernetes.io/projected/a9641f09-cad8-4bde-9109-1a7da203da71-kube-api-access-hmqx8\") pod \"node-ca-ckjp9\" (UID: \"a9641f09-cad8-4bde-9109-1a7da203da71\") " pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:44.814100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.813980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbtv\" (UniqueName: \"kubernetes.io/projected/c2953d14-8846-4a93-9f00-03b4e7eca783-kube-api-access-wtbtv\") pod \"tuned-xrk55\" (UID: \"c2953d14-8846-4a93-9f00-03b4e7eca783\") " pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:44.814100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.813942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktbd\" (UniqueName: \"kubernetes.io/projected/cc6329f8-e3de-4d56-8353-c1be192f862c-kube-api-access-bktbd\") pod \"multus-zq4qp\" (UID: \"cc6329f8-e3de-4d56-8353-c1be192f862c\") " pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.901294 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-config\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-script-lib\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-netns\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-etc-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-node-log\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-node-log\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-netns\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-etc-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-systemd-units\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-var-lib-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-systemd-units\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-log-socket\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrfw\" (UniqueName: \"kubernetes.io/projected/0c826ad3-a5c4-47ee-9412-c87ebb211883-kube-api-access-jgrfw\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-log-socket\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-sys-fs\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfgm\" (UniqueName: \"kubernetes.io/projected/b31b90ef-b78e-40a5-89c7-b36dabbc993d-kube-api-access-2lfgm\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnqrp\" (UniqueName: \"kubernetes.io/projected/d0c388cc-d3a8-4cee-aea7-8eadddae330e-kube-api-access-qnqrp\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-sys-fs\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-device-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.901834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cdb37a7-d306-4fd0-94ff-dc390a119957-iptables-alerter-script\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdb37a7-d306-4fd0-94ff-dc390a119957-host-slash\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0c388cc-d3a8-4cee-aea7-8eadddae330e-hosts-file\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-ovn\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-bin\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpp5x\" (UniqueName: \"kubernetes.io/projected/57a7b3d9-e8ca-40f4-8960-0d522d492162-kube-api-access-cpp5x\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-kubelet\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-config\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-systemd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-systemd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cdb37a7-d306-4fd0-94ff-dc390a119957-host-slash\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovnkube-script-lib\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0c388cc-d3a8-4cee-aea7-8eadddae330e-hosts-file\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-ovn\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-bin\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.901754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-var-lib-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.902677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-kubelet\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-device-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-env-overrides\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kmf\" (UniqueName: \"kubernetes.io/projected/8cdb37a7-d306-4fd0-94ff-dc390a119957-kube-api-access-n4kmf\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-socket-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-netd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0c388cc-d3a8-4cee-aea7-8eadddae330e-tmp-dir\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-registration-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-slash\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.903568 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.902777    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:44.902857    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.402837583 +0000 UTC m=+3.088915696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-slash\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c826ad3-a5c4-47ee-9412-c87ebb211883-env-overrides\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-host-cni-netd\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-registration-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.902974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c826ad3-a5c4-47ee-9412-c87ebb211883-run-openvswitch\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.903005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57a7b3d9-e8ca-40f4-8960-0d522d492162-socket-dir\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.903081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0c388cc-d3a8-4cee-aea7-8eadddae330e-tmp-dir\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.904497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.903087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cdb37a7-d306-4fd0-94ff-dc390a119957-iptables-alerter-script\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.905456 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.905437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c826ad3-a5c4-47ee-9412-c87ebb211883-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.913043 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.912983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfgm\" (UniqueName: \"kubernetes.io/projected/b31b90ef-b78e-40a5-89c7-b36dabbc993d-kube-api-access-2lfgm\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:44.913043 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.913017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrfw\" (UniqueName: \"kubernetes.io/projected/0c826ad3-a5c4-47ee-9412-c87ebb211883-kube-api-access-jgrfw\") pod \"ovnkube-node-9dgs4\" (UID: \"0c826ad3-a5c4-47ee-9412-c87ebb211883\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:44.913251 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.913232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpp5x\" (UniqueName: \"kubernetes.io/projected/57a7b3d9-e8ca-40f4-8960-0d522d492162-kube-api-access-cpp5x\") pod \"aws-ebs-csi-driver-node-wb2rr\" (UID: \"57a7b3d9-e8ca-40f4-8960-0d522d492162\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:44.913338 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.913314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kmf\" (UniqueName: \"kubernetes.io/projected/8cdb37a7-d306-4fd0-94ff-dc390a119957-kube-api-access-n4kmf\") pod \"iptables-alerter-f5rfw\" (UID: \"8cdb37a7-d306-4fd0-94ff-dc390a119957\") " pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:44.913464 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.913438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnqrp\" (UniqueName: \"kubernetes.io/projected/d0c388cc-d3a8-4cee-aea7-8eadddae330e-kube-api-access-qnqrp\") pod \"node-resolver-jntvr\" (UID: \"d0c388cc-d3a8-4cee-aea7-8eadddae330e\") " pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:44.985448 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:44.985404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zq4qp"
Jul 28 14:22:44.993726 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:44.993698    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6329f8_e3de_4d56_8353_c1be192f862c.slice/crio-96aec40a1947a6202bdf143ea47dfcd1cc0ff46f67e6f97af3b82fa86f4d7202 WatchSource:0}: Error finding container 96aec40a1947a6202bdf143ea47dfcd1cc0ff46f67e6f97af3b82fa86f4d7202: Status 404 returned error can't find the container with id 96aec40a1947a6202bdf143ea47dfcd1cc0ff46f67e6f97af3b82fa86f4d7202
Jul 28 14:22:45.002735 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.002712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:22:45.009930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.009910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr"
Jul 28 14:22:45.015190 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.015166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ckjp9"
Jul 28 14:22:45.018308 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.018276    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cdcf6e_c202_43bb_8653_31f0c22b0e5a.slice/crio-9e602032f8389534408593debcada10d5ad0d74fe07062ae2bd853d5e3cfdc62 WatchSource:0}: Error finding container 9e602032f8389534408593debcada10d5ad0d74fe07062ae2bd853d5e3cfdc62: Status 404 returned error can't find the container with id 9e602032f8389534408593debcada10d5ad0d74fe07062ae2bd853d5e3cfdc62
Jul 28 14:22:45.021826 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.021797    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:45.021826 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.021810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrk55"
Jul 28 14:22:45.023527 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.023506    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a7b3d9_e8ca_40f4_8960_0d522d492162.slice/crio-63f35c46e528453141471140885d3bfdd0874fd8b8959b6b04b1cd9793458acb WatchSource:0}: Error finding container 63f35c46e528453141471140885d3bfdd0874fd8b8959b6b04b1cd9793458acb: Status 404 returned error can't find the container with id 63f35c46e528453141471140885d3bfdd0874fd8b8959b6b04b1cd9793458acb
Jul 28 14:22:45.024319 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.024257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9641f09_cad8_4bde_9109_1a7da203da71.slice/crio-2235b50112c047648aa318e9d736ef27a4f00a2d84ba9db499c47c7a5f346782 WatchSource:0}: Error finding container 2235b50112c047648aa318e9d736ef27a4f00a2d84ba9db499c47c7a5f346782: Status 404 returned error can't find the container with id 2235b50112c047648aa318e9d736ef27a4f00a2d84ba9db499c47c7a5f346782
Jul 28 14:22:45.028556 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.028449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fktm"
Jul 28 14:22:45.030346 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.030324    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2953d14_8846_4a93_9f00_03b4e7eca783.slice/crio-927a3957815774f5d6ccdfd0370699d40aa3871425ab0b54beaf31d606f698a4 WatchSource:0}: Error finding container 927a3957815774f5d6ccdfd0370699d40aa3871425ab0b54beaf31d606f698a4: Status 404 returned error can't find the container with id 927a3957815774f5d6ccdfd0370699d40aa3871425ab0b54beaf31d606f698a4
Jul 28 14:22:45.037675 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.037650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:22:45.038073 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.038051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b6f0b0_9666_4042_ab4a_1887720c36ef.slice/crio-8bd9e76bb0c6147129f1e47ff01ec7d8548617e1e1a4c80d736b17109e9b1c97 WatchSource:0}: Error finding container 8bd9e76bb0c6147129f1e47ff01ec7d8548617e1e1a4c80d736b17109e9b1c97: Status 404 returned error can't find the container with id 8bd9e76bb0c6147129f1e47ff01ec7d8548617e1e1a4c80d736b17109e9b1c97
Jul 28 14:22:45.047552 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.047528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f5rfw"
Jul 28 14:22:45.047878 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.047858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c826ad3_a5c4_47ee_9412_c87ebb211883.slice/crio-bc2aa74d5d67d1d6b58845a508d757c68c59ab4c38fc207a8d8e2297d4b1af40 WatchSource:0}: Error finding container bc2aa74d5d67d1d6b58845a508d757c68c59ab4c38fc207a8d8e2297d4b1af40: Status 404 returned error can't find the container with id bc2aa74d5d67d1d6b58845a508d757c68c59ab4c38fc207a8d8e2297d4b1af40
Jul 28 14:22:45.052430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.052407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jntvr"
Jul 28 14:22:45.063803 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:22:45.063773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c388cc_d3a8_4cee_aea7_8eadddae330e.slice/crio-d5b24abd592d3b1275bc875ae9367f36d18e27f85e4b4acaba650e36c360f964 WatchSource:0}: Error finding container d5b24abd592d3b1275bc875ae9367f36d18e27f85e4b4acaba650e36c360f964: Status 404 returned error can't find the container with id d5b24abd592d3b1275bc875ae9367f36d18e27f85e4b4acaba650e36c360f964
Jul 28 14:22:45.306188 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.306154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:45.306325 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.306269    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:45.306365 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.306325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:22:46.306311048 +0000 UTC m=+3.992389142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:45.407162 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.407122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:45.407350 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.407186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:45.407350 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407307    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:45.407350 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407322    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:45.407350 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407344    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:45.407584 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407356    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:45.407584 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407382    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:22:46.407361727 +0000 UTC m=+4.093439834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:45.407584 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:45.407401    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:46.407390583 +0000 UTC m=+4.093468677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:45.736138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.736030    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:17:44 +0000 UTC" deadline="2028-02-13 05:51:33.37856041 +0000 UTC"
Jul 28 14:22:45.736138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.736111    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13551h28m47.642454108s"
Jul 28 14:22:45.798130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.797958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"bc2aa74d5d67d1d6b58845a508d757c68c59ab4c38fc207a8d8e2297d4b1af40"}
Jul 28 14:22:45.804260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.804174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jntvr" event={"ID":"d0c388cc-d3a8-4cee-aea7-8eadddae330e","Type":"ContainerStarted","Data":"d5b24abd592d3b1275bc875ae9367f36d18e27f85e4b4acaba650e36c360f964"}
Jul 28 14:22:45.808649 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.808618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerStarted","Data":"8bd9e76bb0c6147129f1e47ff01ec7d8548617e1e1a4c80d736b17109e9b1c97"}
Jul 28 14:22:45.821266 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.820997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrk55" event={"ID":"c2953d14-8846-4a93-9f00-03b4e7eca783","Type":"ContainerStarted","Data":"927a3957815774f5d6ccdfd0370699d40aa3871425ab0b54beaf31d606f698a4"}
Jul 28 14:22:45.848328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.848281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckjp9" event={"ID":"a9641f09-cad8-4bde-9109-1a7da203da71","Type":"ContainerStarted","Data":"2235b50112c047648aa318e9d736ef27a4f00a2d84ba9db499c47c7a5f346782"}
Jul 28 14:22:45.863025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.862919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr" event={"ID":"57a7b3d9-e8ca-40f4-8960-0d522d492162","Type":"ContainerStarted","Data":"63f35c46e528453141471140885d3bfdd0874fd8b8959b6b04b1cd9793458acb"}
Jul 28 14:22:45.867889 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.867547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgnhf" event={"ID":"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a","Type":"ContainerStarted","Data":"9e602032f8389534408593debcada10d5ad0d74fe07062ae2bd853d5e3cfdc62"}
Jul 28 14:22:45.879215 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.879178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zq4qp" event={"ID":"cc6329f8-e3de-4d56-8353-c1be192f862c","Type":"ContainerStarted","Data":"96aec40a1947a6202bdf143ea47dfcd1cc0ff46f67e6f97af3b82fa86f4d7202"}
Jul 28 14:22:45.897082 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.897046    2577 generic.go:356] "Generic (PLEG): container finished" podID="1c227a17b6b32368aeb98bed0f11f5ed" containerID="a7781f4f479c8d52c4beaa76ee448277a1d44f6aaece3a0a035eda7a2971f21e" exitCode=0
Jul 28 14:22:45.897385 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.897353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"1c227a17b6b32368aeb98bed0f11f5ed","Type":"ContainerDied","Data":"a7781f4f479c8d52c4beaa76ee448277a1d44f6aaece3a0a035eda7a2971f21e"}
Jul 28 14:22:45.909059 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:45.909015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5rfw" event={"ID":"8cdb37a7-d306-4fd0-94ff-dc390a119957","Type":"ContainerStarted","Data":"dee252351a8040ed092ee3d0bf425e7f0c63176594d8d0393c6bc44fc5b40696"}
Jul 28 14:22:46.316631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.316348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:46.316842 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.316788    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:46.316914 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.316860    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:22:48.316839479 +0000 UTC m=+6.002917579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:46.417545 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.417420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:46.417545 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.417502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:46.417801 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.417676    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:46.417801 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.417695    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:46.417801 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.417709    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:46.417801 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.417770    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:48.417750574 +0000 UTC m=+6.103828671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:46.418235 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.418209    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:46.418318 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.418264    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:22:48.418249397 +0000 UTC m=+6.104327496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:46.521178 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.520305    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:46.737130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.737020    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:17:44 +0000 UTC" deadline="2028-01-11 02:44:10.393688144 +0000 UTC"
Jul 28 14:22:46.737130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.737074    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12756h21m23.656617119s"
Jul 28 14:22:46.786068 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.785841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:46.786068 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.785975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:46.787111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.786397    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:46.787111 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.786500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:46.787111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.786573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:46.787111 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:46.786678    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:46.928361 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.928319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" event={"ID":"1c227a17b6b32368aeb98bed0f11f5ed","Type":"ContainerStarted","Data":"7e0f3706cdcf6387e19e6939342bdf8f3217eb9d2a1d37d0a7b4334bbd0a560e"}
Jul 28 14:22:46.947432 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:46.947208    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-230.ec2.internal" podStartSLOduration=2.947185157 podStartE2EDuration="2.947185157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:46.946168706 +0000 UTC m=+4.632246825" watchObservedRunningTime="2026-07-28 14:22:46.947185157 +0000 UTC m=+4.633263276"
Jul 28 14:22:48.332810 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.332777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:48.333280 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.332966    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:48.333280 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.333046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:22:52.333026622 +0000 UTC m=+10.019104715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:48.434028 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.433990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:48.434192 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.434072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:48.434192 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434146    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:48.434313 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434216    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:48.434313 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:22:52.434200762 +0000 UTC m=+10.120278865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:48.434313 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434232    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:48.434313 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434245    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:48.434313 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.434292    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:52.434278079 +0000 UTC m=+10.120356190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:48.780655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.780625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:48.780655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.780653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:48.780880 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:48.780666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:48.780880 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.780758    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:48.781205 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.781179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:48.781303 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:48.781267    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:50.780139 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:50.780098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:50.780595 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:50.780244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:50.780595 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:50.780305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:50.780595 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:50.780422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:50.780595 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:50.780464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:50.780595 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:50.780545    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:52.364386 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.364235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:52.364825 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.364405    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:52.364825 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.364481    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:23:00.364460596 +0000 UTC m=+18.050538688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:52.465127 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.465085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:52.465292 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.465161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:52.465292 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465287    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:52.465411 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465335    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:52.465411 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465361    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:52.465411 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465375    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:52.465411 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:23:00.465329064 +0000 UTC m=+18.151407161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:52.465633 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.465436    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:00.465419994 +0000 UTC m=+18.151498090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.780636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.780761    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.781116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.781200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:52.781250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:52.781370 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:52.781332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:54.780085 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:54.780048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:54.780560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:54.780048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:54.780560 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:54.780187    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:54.780560 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:54.780277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:54.780560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:54.780060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:54.780560 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:54.780374    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:56.780202 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:56.780166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:56.780683 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:56.780289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:56.780683 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:56.780303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:56.780683 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:56.780428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:22:56.780683 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:56.780478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:56.780683 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:56.780570    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:58.780372 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:58.780333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:22:58.780831 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:58.780333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:22:58.780831 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:58.780459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:22:58.780831 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:58.780561    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:22:58.780831 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:22:58.780333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:22:58.780831 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:22:58.780670    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:00.420604 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.420515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:00.421015 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.420656    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:00.421015 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.420717    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:23:16.420701369 +0000 UTC m=+34.106779464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:00.521733 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.521691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:00.521902 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.521762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:00.521902 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521862    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:00.522021 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521893    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:23:00.522021 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521919    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:23:00.522021 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521933    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:00.522021 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521938    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:23:16.521918019 +0000 UTC m=+34.207996113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:00.522021 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.521981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:16.521966151 +0000 UTC m=+34.208044246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:00.779992 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.779962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:00.779992 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.779992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:00.780231 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:00.779968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:00.780231 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.780082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:00.780231 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.780148    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:00.780385 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:00.780257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:02.780957 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:02.780917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:02.781369 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:02.781005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:02.781369 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:02.781045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:02.781369 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:02.781064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:02.781369 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:02.781122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:02.781369 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:02.781215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:04.779988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.779738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:04.780651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.779747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:04.780651 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:04.780039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:04.780651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.779785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:04.780651 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:04.780090    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:04.780651 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:04.780183    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:04.966566 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.966519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"3500f14e55c29fac4c1c829670e3f94a52e52bf86526f6e305ca3de816a38205"}
Jul 28 14:23:04.967951 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.967930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" event={"ID":"206bac433f2fce4d8e4d8f2249cfb20b","Type":"ContainerStarted","Data":"79aa39abbbdaaac19fe5c76bf46b1037e4747694bb844f568b42c0aefaf8c25d"}
Jul 28 14:23:04.969248 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.969227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jntvr" event={"ID":"d0c388cc-d3a8-4cee-aea7-8eadddae330e","Type":"ContainerStarted","Data":"bf5aaec3f89cad2f76b081600337701548661107bb1c5e809364f86310cdbeb8"}
Jul 28 14:23:04.970692 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.970669    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="0e482b57ad6fbddbf8f23e475d882fd30beecef1657b2ef63fc5fcc5adfd2e69" exitCode=0
Jul 28 14:23:04.970778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.970747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"0e482b57ad6fbddbf8f23e475d882fd30beecef1657b2ef63fc5fcc5adfd2e69"}
Jul 28 14:23:04.972203 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.972181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrk55" event={"ID":"c2953d14-8846-4a93-9f00-03b4e7eca783","Type":"ContainerStarted","Data":"56db73d15a8d3c5b4aae7b5843a6aedcf7a3beee6237964a12fb91adbcbf2a96"}
Jul 28 14:23:04.973636 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.973618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ckjp9" event={"ID":"a9641f09-cad8-4bde-9109-1a7da203da71","Type":"ContainerStarted","Data":"e1ee4093882b13f386a3ddb6d8fd76a799d6f1eeea5af40cdacaa577c810b7a2"}
Jul 28 14:23:04.974957 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.974940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr" event={"ID":"57a7b3d9-e8ca-40f4-8960-0d522d492162","Type":"ContainerStarted","Data":"acaa3f439c4a1e131b21e295060d3a1637fcfc521f7e7caa18caf7a222a55130"}
Jul 28 14:23:04.976372 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.976348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgnhf" event={"ID":"c0cdcf6e-c202-43bb-8653-31f0c22b0e5a","Type":"ContainerStarted","Data":"e64dfca2c4761e965df30f9b2d89b9b040cc3a03ada72f45ea8db7e2aac78e32"}
Jul 28 14:23:04.977784 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:04.977763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zq4qp" event={"ID":"cc6329f8-e3de-4d56-8353-c1be192f862c","Type":"ContainerStarted","Data":"b5a1ae033de6110ff2b7917db38cb34d626d585eaf8c834984b5abba7ac789b5"}
Jul 28 14:23:05.006505 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.006453    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-230.ec2.internal" podStartSLOduration=21.006431431 podStartE2EDuration="21.006431431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:04.98307838 +0000 UTC m=+22.669156494" watchObservedRunningTime="2026-07-28 14:23:05.006431431 +0000 UTC m=+22.692509549"
Jul 28 14:23:05.041759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.041709    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xrk55" podStartSLOduration=2.718886245 podStartE2EDuration="22.041693974s" totalImagesPullingTime="19.322807729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:05.024596921 +0000 UTC m=+22.710675035" watchObservedRunningTime="2026-07-28 14:23:05.041693974 +0000 UTC m=+22.727772086"
Jul 28 14:23:05.041928 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.041838    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jntvr" podStartSLOduration=2.751476041 podStartE2EDuration="22.041829477s" totalImagesPullingTime="19.290353436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:05.041494289 +0000 UTC m=+22.727572405" watchObservedRunningTime="2026-07-28 14:23:05.041829477 +0000 UTC m=+22.727907593"
Jul 28 14:23:05.063964 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.063912    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zq4qp" podStartSLOduration=2.703765632 podStartE2EDuration="22.063896516s" totalImagesPullingTime="19.360130884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:05.063425404 +0000 UTC m=+22.749503519" watchObservedRunningTime="2026-07-28 14:23:05.063896516 +0000 UTC m=+22.749974653"
Jul 28 14:23:05.079701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.079657    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgnhf" podStartSLOduration=7.509533412 podStartE2EDuration="22.079640362s" totalImagesPullingTime="14.57010695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:05.079464206 +0000 UTC m=+22.765542321" watchObservedRunningTime="2026-07-28 14:23:05.079640362 +0000 UTC m=+22.765718475"
Jul 28 14:23:05.096553 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.096358    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ckjp9" podStartSLOduration=2.8572290970000003 podStartE2EDuration="22.096339089s" totalImagesPullingTime="19.239109992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:05.095601327 +0000 UTC m=+22.781679442" watchObservedRunningTime="2026-07-28 14:23:05.096339089 +0000 UTC m=+22.782417199"
Jul 28 14:23:05.664067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.664043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:23:05.664957 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.664935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:23:05.800685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.800659    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 14:23:05.982617 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.982582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr" event={"ID":"57a7b3d9-e8ca-40f4-8960-0d522d492162","Type":"ContainerStarted","Data":"6d1a42fcb0b2543b86d27105ba9c5100e6136a26fbfee65542982f82c0598d66"}
Jul 28 14:23:05.984177 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.984127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f5rfw" event={"ID":"8cdb37a7-d306-4fd0-94ff-dc390a119957","Type":"ContainerStarted","Data":"7c1bb69f13c5aaf656fa8e3b0efa8f94d329a15f6383d7f610b63804e44745b0"}
Jul 28 14:23:05.987114 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987097    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:23:05.987451 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987433    2577 generic.go:356] "Generic (PLEG): container finished" podID="0c826ad3-a5c4-47ee-9412-c87ebb211883" containerID="edbf5efbf09dd85e61ef4557c9a63e0aa369f44722f905e337760bf236ae5b95" exitCode=1
Jul 28 14:23:05.987551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerDied","Data":"edbf5efbf09dd85e61ef4557c9a63e0aa369f44722f905e337760bf236ae5b95"}
Jul 28 14:23:05.987616 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"c53eb3cede82501c08f36818837f17b7e82cd42a275388f6f33d90ea4c9ba43b"}
Jul 28 14:23:05.987668 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"a4f9097eee3f3459e2f19d86bad5735880321dcf36792ef0acc3474c72fba99f"}
Jul 28 14:23:05.987722 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"3b7b7b486b25a0138c7d1403ad3626a6578c17f6da339ef6f4a120d693b45213"}
Jul 28 14:23:05.987772 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:05.987760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"3d55159a41cd25e69efeff8f47b7e72690abfbbf830eb0685bf55dfc41ce7f9d"}
Jul 28 14:23:06.764336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.764222    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T14:23:05.800678614Z","UUID":"c0048911-6acd-4577-9667-f5b3af3f02f1","Handler":null,"Name":"","Endpoint":""}
Jul 28 14:23:06.766118 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.766094    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 14:23:06.766247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.766133    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 14:23:06.780308 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.780281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:06.780454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.780281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:06.780454 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:06.780409    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:06.780454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.780426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:06.780641 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:06.780521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:06.780641 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:06.780613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:06.989551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:06.989518    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 14:23:07.357081 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.357045    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:23:07.357771 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.357749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgnhf"
Jul 28 14:23:07.376104 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.374719    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f5rfw" podStartSLOduration=9.45934853 podStartE2EDuration="24.374699182s" totalImagesPullingTime="14.915350652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:06.008606298 +0000 UTC m=+23.694684413" watchObservedRunningTime="2026-07-28 14:23:07.374699182 +0000 UTC m=+25.060777295"
Jul 28 14:23:07.995636 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.995408    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:23:07.996097 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.996072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"d6bd5864a3af732aa8ac7571f7b5158aaf9a287f96615d95bde6a5aaad071254"}
Jul 28 14:23:07.998644 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:07.998617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr" event={"ID":"57a7b3d9-e8ca-40f4-8960-0d522d492162","Type":"ContainerStarted","Data":"411cfd144d28c16af3788ba054cbe27154bceee4e72ad06e3c7534f058a83b2f"}
Jul 28 14:23:08.779797 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:08.779757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:08.780008 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:08.779855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:08.780008 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:08.779953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:08.780008 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:08.779965    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:08.780165 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:08.780096    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:08.780218 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:08.780197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:10.007239 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.007062    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:23:10.007986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.007570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"56c3f6d2bb2e226c0476b0ef8a4e258178d3499f6d2a799dda42fd98cd649dec"}
Jul 28 14:23:10.007986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.007905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:10.007986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.007972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:10.008140 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.008110    2577 scope.go:117] "RemoveContainer" containerID="edbf5efbf09dd85e61ef4557c9a63e0aa369f44722f905e337760bf236ae5b95"
Jul 28 14:23:10.009742 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.009717    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="b3b3cedb98d1ea796d084975ff9ed9b992be2eadaff7e3e9013aedd616b05b99" exitCode=0
Jul 28 14:23:10.009855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.009765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"b3b3cedb98d1ea796d084975ff9ed9b992be2eadaff7e3e9013aedd616b05b99"}
Jul 28 14:23:10.025034 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.025011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:10.038796 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.038755    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb2rr" podStartSLOduration=5.598420406 podStartE2EDuration="27.038740135s" totalImagesPullingTime="21.440319729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:08.017147674 +0000 UTC m=+25.703225788" watchObservedRunningTime="2026-07-28 14:23:10.038740135 +0000 UTC m=+27.724818309"
Jul 28 14:23:10.779988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.779949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:10.779988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.779967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:10.780259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:10.779950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:10.780259 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:10.780080    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:10.780259 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:10.780219    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:10.780400 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:10.780316    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:11.015930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.015907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:23:11.016342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.016230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" event={"ID":"0c826ad3-a5c4-47ee-9412-c87ebb211883","Type":"ContainerStarted","Data":"0c7b26a650b7e0d515361f9552887b87c1f10affcbc533d57f613b3a614725d3"}
Jul 28 14:23:11.016554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.016519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:11.031865 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.031841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:11.089551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.089480    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4" podStartSLOduration=8.350809257 podStartE2EDuration="28.089462927s" totalImagesPullingTime="19.73865367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:11.052236607 +0000 UTC m=+28.738314747" watchObservedRunningTime="2026-07-28 14:23:11.089462927 +0000 UTC m=+28.775541040"
Jul 28 14:23:11.810355 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.810322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fzdm"]
Jul 28 14:23:11.810571 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.810477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:11.810638 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:11.810588    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:11.814895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.814867    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqcgv"]
Jul 28 14:23:11.815057 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.814981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:11.815124 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:11.815089    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:11.815450 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.815431    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2n29"]
Jul 28 14:23:11.815557 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:11.815525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:11.815651 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:11.815633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:12.021171 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:12.020995    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="3212a6d6b35006674a401d50172589eff2924a7eb1d2a09c79f743cf95b43c83" exitCode=0
Jul 28 14:23:12.021567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:12.021082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"3212a6d6b35006674a401d50172589eff2924a7eb1d2a09c79f743cf95b43c83"}
Jul 28 14:23:13.779995 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:13.779971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:13.780355 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:13.780125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:13.780355 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:13.780151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:13.780355 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:13.780142    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:13.780355 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:13.780211    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:13.780355 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:13.780274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:14.027435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:14.027352    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="c7ad4cea55e997d8a7d435bad8e3986e8ea88f8d7919760067f008a83df02ab6" exitCode=0
Jul 28 14:23:14.027435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:14.027406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"c7ad4cea55e997d8a7d435bad8e3986e8ea88f8d7919760067f008a83df02ab6"}
Jul 28 14:23:15.780412 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:15.780381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:15.781101 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:15.780435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:15.781101 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:15.780656    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:15.781101 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:15.780805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:15.781101 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:15.780843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:15.781101 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:15.780912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:16.447764 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:16.447728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:16.447933 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.447908    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:16.448011 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.447989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret podName:3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b nodeName:}" failed. No retries permitted until 2026-07-28 14:23:48.447967712 +0000 UTC m=+66.134045822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret") pod "global-pull-secret-syncer-4fzdm" (UID: "3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:16.548274 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:16.548195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:16.548274 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:16.548267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548371    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548387    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548404    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548419    2577 projected.go:196] Error preparing data for projected volume kube-api-access-x292v for pod openshift-network-diagnostics/network-check-target-x2n29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548443    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:23:48.548426649 +0000 UTC m=+66.234504740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:16.548484 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:16.548465    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v podName:c528e538-28d8-4104-aa67-db7f368d07e2 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:48.548450347 +0000 UTC m=+66.234528453 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x292v" (UniqueName: "kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v") pod "network-check-target-x2n29" (UID: "c528e538-28d8-4104-aa67-db7f368d07e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:17.779786 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:17.779752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:17.780306 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:17.779762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:17.780306 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:17.779877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:17.780306 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:17.779879    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x2n29" podUID="c528e538-28d8-4104-aa67-db7f368d07e2"
Jul 28 14:23:17.780306 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:17.779976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4fzdm" podUID="3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b"
Jul 28 14:23:17.780306 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:17.780042    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqcgv" podUID="b31b90ef-b78e-40a5-89c7-b36dabbc993d"
Jul 28 14:23:18.050247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.050218    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-230.ec2.internal" event="NodeReady"
Jul 28 14:23:18.050386 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.050374    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 14:23:18.097621 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.097592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9chw8"]
Jul 28 14:23:18.099389 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.099373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.102673 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.102651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 14:23:18.102791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.102651    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 14:23:18.102791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.102656    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl2dt\""
Jul 28 14:23:18.102938 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.102915    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m7z6c"]
Jul 28 14:23:18.104892 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.104873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.107660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.107265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-htnvc\""
Jul 28 14:23:18.107660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.107498    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 14:23:18.107660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.107588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 14:23:18.107821 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.107736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 14:23:18.121277 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.121243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9chw8"]
Jul 28 14:23:18.121800 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.121774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7z6c"]
Jul 28 14:23:18.257524 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3529d4aa-3f68-4339-824d-dbfc4872c9ce-tmp-dir\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.257680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.257680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwcg\" (UniqueName: \"kubernetes.io/projected/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-kube-api-access-2zwcg\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.257785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh4j7\" (UniqueName: \"kubernetes.io/projected/3529d4aa-3f68-4339-824d-dbfc4872c9ce-kube-api-access-jh4j7\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.257785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-cert\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.257871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.257784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3529d4aa-3f68-4339-824d-dbfc4872c9ce-config-volume\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.358838 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.358752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.358838 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.358792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwcg\" (UniqueName: \"kubernetes.io/projected/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-kube-api-access-2zwcg\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.358838 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.358819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh4j7\" (UniqueName: \"kubernetes.io/projected/3529d4aa-3f68-4339-824d-dbfc4872c9ce-kube-api-access-jh4j7\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.359137 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.358943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-cert\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.359137 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.358985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3529d4aa-3f68-4339-824d-dbfc4872c9ce-config-volume\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.359137 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.359031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3529d4aa-3f68-4339-824d-dbfc4872c9ce-tmp-dir\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.359137 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:18.359045    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:18.359137 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:18.359113    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:18.859091706 +0000 UTC m=+36.545169797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:18.359367 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.359323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3529d4aa-3f68-4339-824d-dbfc4872c9ce-tmp-dir\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.359629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.359610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3529d4aa-3f68-4339-824d-dbfc4872c9ce-config-volume\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.363230 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.363202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-cert\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.369629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.369462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh4j7\" (UniqueName: \"kubernetes.io/projected/3529d4aa-3f68-4339-824d-dbfc4872c9ce-kube-api-access-jh4j7\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.369718 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.369666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwcg\" (UniqueName: \"kubernetes.io/projected/958f61c6-5bc6-42e7-ac00-c6e86ec3d35c-kube-api-access-2zwcg\") pod \"ingress-canary-m7z6c\" (UID: \"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c\") " pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.415735 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.415701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m7z6c"
Jul 28 14:23:18.545869 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.545829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m7z6c"]
Jul 28 14:23:18.549416 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:23:18.549389    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958f61c6_5bc6_42e7_ac00_c6e86ec3d35c.slice/crio-3ca0478fcbb2ae5da217b4d63fa50cde4dbc83fdb76b87dd6ec7a76265fd403a WatchSource:0}: Error finding container 3ca0478fcbb2ae5da217b4d63fa50cde4dbc83fdb76b87dd6ec7a76265fd403a: Status 404 returned error can't find the container with id 3ca0478fcbb2ae5da217b4d63fa50cde4dbc83fdb76b87dd6ec7a76265fd403a
Jul 28 14:23:18.861784 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:18.861749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:18.862282 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:18.861939    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:18.862282 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:18.862043    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:19.862021628 +0000 UTC m=+37.548099725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:19.044624 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.044589    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="6a1e90ba6b5be29b520f5f01da7919bd500886e36c5e1bacdeacf29d2ab3cc5a" exitCode=0
Jul 28 14:23:19.044792 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.044670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"6a1e90ba6b5be29b520f5f01da7919bd500886e36c5e1bacdeacf29d2ab3cc5a"}
Jul 28 14:23:19.046150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.046125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7z6c" event={"ID":"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c","Type":"ContainerStarted","Data":"3ca0478fcbb2ae5da217b4d63fa50cde4dbc83fdb76b87dd6ec7a76265fd403a"}
Jul 28 14:23:19.780061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.780023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:19.780249 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.780023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:19.780518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.780032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:19.783875 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.783714    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 14:23:19.783875 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.783754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 14:23:19.783875 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.783754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kq7zv\""
Jul 28 14:23:19.783875 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.783753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z6ktw\""
Jul 28 14:23:19.784201 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.784183    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 14:23:19.784481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.784463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 14:23:19.867338 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:19.867297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:19.867724 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:19.867411    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:19.867724 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:19.867476    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:21.867462895 +0000 UTC m=+39.553540986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:20.051569 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:20.051459    2577 generic.go:356] "Generic (PLEG): container finished" podID="99b6f0b0-9666-4042-ab4a-1887720c36ef" containerID="b85068a18ae919074cec2f25a81e7fd16d949771749e097fbd8fc7d5b511cd0c" exitCode=0
Jul 28 14:23:20.051569 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:20.051518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerDied","Data":"b85068a18ae919074cec2f25a81e7fd16d949771749e097fbd8fc7d5b511cd0c"}
Jul 28 14:23:21.058760 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:21.057882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fktm" event={"ID":"99b6f0b0-9666-4042-ab4a-1887720c36ef","Type":"ContainerStarted","Data":"01d0265456ee31f5fd23e22c449831a8f837eb21295fb560a478fd2070470266"}
Jul 28 14:23:21.060157 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:21.060127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m7z6c" event={"ID":"958f61c6-5bc6-42e7-ac00-c6e86ec3d35c","Type":"ContainerStarted","Data":"17140ef818ce146207c1f845da1ebc403ad79e94e7be5a3e852905e3de183c1f"}
Jul 28 14:23:21.086040 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:21.085983    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5fktm" podStartSLOduration=8.224047965 podStartE2EDuration="38.085962593s" totalImagesPullingTime="29.861914628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:21.084762645 +0000 UTC m=+38.770840758" watchObservedRunningTime="2026-07-28 14:23:21.085962593 +0000 UTC m=+38.772040711"
Jul 28 14:23:21.103192 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:21.103142    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m7z6c" podStartSLOduration=0.831245669 podStartE2EDuration="3.103128275s" totalImagesPullingTime="2.271882606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:21.10268269 +0000 UTC m=+38.788760802" watchObservedRunningTime="2026-07-28 14:23:21.103128275 +0000 UTC m=+38.789206379"
Jul 28 14:23:21.884738 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:21.884694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:21.884898 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:21.884813    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:21.884898 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:21.884866    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:25.88485415 +0000 UTC m=+43.570932241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:25.911858 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:25.911811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:25.912334 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:25.911965    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:25.912334 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:25.912033    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:33.912017423 +0000 UTC m=+51.598095514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:33.965833 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:33.965794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:33.966293 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:33.965919    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:33.966293 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:33.965984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:23:49.965968892 +0000 UTC m=+67.652047000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:43.052441 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:43.052410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgs4"
Jul 28 14:23:48.465846 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.465787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:48.468658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.468636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 14:23:48.480184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.480162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b-original-pull-secret\") pod \"global-pull-secret-syncer-4fzdm\" (UID: \"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b\") " pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:48.566918 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.566877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:23:48.567096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.566931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:48.569605 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.569585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 14:23:48.569663 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.569609    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 14:23:48.577520 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:48.577500    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 14:23:48.577610 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:48.577599    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs podName:b31b90ef-b78e-40a5-89c7-b36dabbc993d nodeName:}" failed. No retries permitted until 2026-07-28 14:24:52.577581099 +0000 UTC m=+130.263659190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs") pod "network-metrics-daemon-sqcgv" (UID: "b31b90ef-b78e-40a5-89c7-b36dabbc993d") : secret "metrics-daemon-secret" not found
Jul 28 14:23:48.579629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.579614    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 14:23:48.590787 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.590763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x292v\" (UniqueName: \"kubernetes.io/projected/c528e538-28d8-4104-aa67-db7f368d07e2-kube-api-access-x292v\") pod \"network-check-target-x2n29\" (UID: \"c528e538-28d8-4104-aa67-db7f368d07e2\") " pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:48.590867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.590855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4fzdm"
Jul 28 14:23:48.606579 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.606560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kq7zv\""
Jul 28 14:23:48.614679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.614660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:48.727028 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.726964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4fzdm"]
Jul 28 14:23:48.733963 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:23:48.733938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe758b3_c1a0_4e29_a50f_ce4b9c06dd8b.slice/crio-090c2072a06b2e895facdc8223b99faadd79f630359af76b6b914dd0d25e0c98 WatchSource:0}: Error finding container 090c2072a06b2e895facdc8223b99faadd79f630359af76b6b914dd0d25e0c98: Status 404 returned error can't find the container with id 090c2072a06b2e895facdc8223b99faadd79f630359af76b6b914dd0d25e0c98
Jul 28 14:23:48.747517 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:48.747493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x2n29"]
Jul 28 14:23:48.749927 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:23:48.749904    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc528e538_28d8_4104_aa67_db7f368d07e2.slice/crio-bbaa5f4ab16ea3e714cbe08e8cbf11639908bf8ab2921191b4e04b3d33c29b20 WatchSource:0}: Error finding container bbaa5f4ab16ea3e714cbe08e8cbf11639908bf8ab2921191b4e04b3d33c29b20: Status 404 returned error can't find the container with id bbaa5f4ab16ea3e714cbe08e8cbf11639908bf8ab2921191b4e04b3d33c29b20
Jul 28 14:23:49.126927 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:49.126854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2n29" event={"ID":"c528e538-28d8-4104-aa67-db7f368d07e2","Type":"ContainerStarted","Data":"bbaa5f4ab16ea3e714cbe08e8cbf11639908bf8ab2921191b4e04b3d33c29b20"}
Jul 28 14:23:49.128192 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:49.128164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fzdm" event={"ID":"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b","Type":"ContainerStarted","Data":"090c2072a06b2e895facdc8223b99faadd79f630359af76b6b914dd0d25e0c98"}
Jul 28 14:23:49.975026 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:49.974987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:23:49.975596 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:49.975163    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:49.975596 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:23:49.975224    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:24:21.975206481 +0000 UTC m=+99.661284573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:54.143658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:54.143626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x2n29" event={"ID":"c528e538-28d8-4104-aa67-db7f368d07e2","Type":"ContainerStarted","Data":"1b104df93de48d68d50c23c884285de41bed577a9ef441e3b82265b1c5704ac4"}
Jul 28 14:23:54.144121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:54.143740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:23:54.145003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:54.144975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4fzdm" event={"ID":"3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b","Type":"ContainerStarted","Data":"48eb4a2924dc1b952ccf14dbf62d95ad2d318ebbd5d0e65140af88e633d6c5cb"}
Jul 28 14:23:54.159949 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:54.159906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x2n29" podStartSLOduration=66.740667874 podStartE2EDuration="1m11.159893756s" totalImagesPullingTime="4.419225882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:54.159322549 +0000 UTC m=+71.845400662" watchObservedRunningTime="2026-07-28 14:23:54.159893756 +0000 UTC m=+71.845971869"
Jul 28 14:23:54.174282 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:23:54.174242    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4fzdm" podStartSLOduration=66.735183179 podStartE2EDuration="1m11.174228596s" totalImagesPullingTime="4.439045417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:54.173918658 +0000 UTC m=+71.859996776" watchObservedRunningTime="2026-07-28 14:23:54.174228596 +0000 UTC m=+71.860306709"
Jul 28 14:24:09.563069 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.562928    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-757957ffc-frc2z"]
Jul 28 14:24:09.565431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.565409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.568247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f54wr\""
Jul 28 14:24:09.568247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 14:24:09.568403 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 14:24:09.568696 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568677    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 14:24:09.568696 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568692    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 14:24:09.568878 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.568760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 14:24:09.569247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.569233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 14:24:09.599151 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.599119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-757957ffc-frc2z"]
Jul 28 14:24:09.611954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.611920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxj6g\" (UniqueName: \"kubernetes.io/projected/3300c232-216d-424b-9ce2-7220f6dbf8a3-kube-api-access-mxj6g\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.611954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.611951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-stats-auth\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.612136 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.611975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-default-certificate\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.612136 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.611994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.612136 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.612049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.677812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.677773    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2mw5m"]
Jul 28 14:24:09.679710 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.679688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.682329 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.682308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:24:09.682458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.682339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rzt9r\""
Jul 28 14:24:09.682458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.682351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 14:24:09.682458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.682348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 14:24:09.682458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.682316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 14:24:09.688444 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.688424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 14:24:09.693394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.693372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2mw5m"]
Jul 28 14:24:09.712707 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxj6g\" (UniqueName: \"kubernetes.io/projected/3300c232-216d-424b-9ce2-7220f6dbf8a3-kube-api-access-mxj6g\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.712850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-trusted-ca\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.712850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-stats-auth\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.712850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-default-certificate\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.712850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.712850 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-serving-cert\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:09.712853    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:09.712907    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:10.212889543 +0000 UTC m=+87.898967640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rgc\" (UniqueName: \"kubernetes.io/projected/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-kube-api-access-x6rgc\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.712979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-config\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.713093 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:09.713011    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:10.212994363 +0000 UTC m=+87.899072466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:09.715138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.715119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-default-certificate\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.715230 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.715140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-stats-auth\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.721327 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.721305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxj6g\" (UniqueName: \"kubernetes.io/projected/3300c232-216d-424b-9ce2-7220f6dbf8a3-kube-api-access-mxj6g\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:09.774751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.774723    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"]
Jul 28 14:24:09.776562 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.776529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.779013 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.778990    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 14:24:09.779136 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.779020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p9c7r\""
Jul 28 14:24:09.779518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.779501    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:24:09.779624 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.779609    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 14:24:09.783505 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.783486    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-skv6z"]
Jul 28 14:24:09.785200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.785182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.789766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.789748    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 14:24:09.789895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.789761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 14:24:09.789895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.789891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h5gth\""
Jul 28 14:24:09.790062 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.789763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 14:24:09.790935 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.790918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 14:24:09.797423 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.797395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"]
Jul 28 14:24:09.797671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.797650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 14:24:09.811549 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.811510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-skv6z"]
Jul 28 14:24:09.813877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-trusted-ca\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.813877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf9g\" (UniqueName: \"kubernetes.io/projected/ddbd6182-8e34-4f55-9518-56d442d0c058-kube-api-access-rrf9g\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-serving-cert\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.814018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-service-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.813987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-tmp\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814199 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddbd6182-8e34-4f55-9518-56d442d0c058-serving-cert\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814199 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rgc\" (UniqueName: \"kubernetes.io/projected/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-kube-api-access-x6rgc\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.814199 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.814199 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2ll\" (UniqueName: \"kubernetes.io/projected/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-kube-api-access-dt2ll\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.814199 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-config\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.814435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-snapshots\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.814782 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-config\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.814925 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.814904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-trusted-ca\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.816328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.816305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-serving-cert\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.828510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.828486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rgc\" (UniqueName: \"kubernetes.io/projected/ef95864b-dd12-48f1-b4f7-69eb0e2d0f10-kube-api-access-x6rgc\") pod \"console-operator-584dcb6b54-2mw5m\" (UID: \"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10\") " pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:09.914690 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-snapshots\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.914870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf9g\" (UniqueName: \"kubernetes.io/projected/ddbd6182-8e34-4f55-9518-56d442d0c058-kube-api-access-rrf9g\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.914870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.914959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-service-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.914959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-tmp\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.914959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddbd6182-8e34-4f55-9518-56d442d0c058-serving-cert\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.915109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.914992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.915109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.915036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2ll\" (UniqueName: \"kubernetes.io/projected/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-kube-api-access-dt2ll\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.915215 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:09.915120    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 14:24:09.915215 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:09.915180    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls podName:1f8fec6e-7ff9-4c0c-b177-c7c5893549a4 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:10.415162391 +0000 UTC m=+88.101240499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls") pod "cluster-samples-operator-6748489656-9mvr7" (UID: "1f8fec6e-7ff9-4c0c-b177-c7c5893549a4") : secret "samples-operator-tls" not found
Jul 28 14:24:09.915322 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.915293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-tmp\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.915390 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.915326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ddbd6182-8e34-4f55-9518-56d442d0c058-snapshots\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.915451 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.915408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-service-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.915555 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.915522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddbd6182-8e34-4f55-9518-56d442d0c058-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.917308 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.917291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ddbd6182-8e34-4f55-9518-56d442d0c058-serving-cert\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.924077 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.924045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2ll\" (UniqueName: \"kubernetes.io/projected/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-kube-api-access-dt2ll\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:09.924182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.924083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf9g\" (UniqueName: \"kubernetes.io/projected/ddbd6182-8e34-4f55-9518-56d442d0c058-kube-api-access-rrf9g\") pod \"insights-operator-5885fdb759-skv6z\" (UID: \"ddbd6182-8e34-4f55-9518-56d442d0c058\") " pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:09.988003 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:09.987969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:10.094049 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.093973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-skv6z"
Jul 28 14:24:10.117691 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.117651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2mw5m"]
Jul 28 14:24:10.120404 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:10.120379    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef95864b_dd12_48f1_b4f7_69eb0e2d0f10.slice/crio-b876c7a173be2d8d5dca880dd0158dccf84f81570776e91f804daf4abd02f1cf WatchSource:0}: Error finding container b876c7a173be2d8d5dca880dd0158dccf84f81570776e91f804daf4abd02f1cf: Status 404 returned error can't find the container with id b876c7a173be2d8d5dca880dd0158dccf84f81570776e91f804daf4abd02f1cf
Jul 28 14:24:10.185023 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.184991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" event={"ID":"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10","Type":"ContainerStarted","Data":"b876c7a173be2d8d5dca880dd0158dccf84f81570776e91f804daf4abd02f1cf"}
Jul 28 14:24:10.217465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.217428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:10.217704 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.217482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:10.217704 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:10.217598    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:10.217704 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:10.217673    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:11.217651098 +0000 UTC m=+88.903729195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:10.217704 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:10.217700    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:11.217690012 +0000 UTC m=+88.903768105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:10.218112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.218095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-skv6z"]
Jul 28 14:24:10.220661 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:10.220634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbd6182_8e34_4f55_9518_56d442d0c058.slice/crio-53f46880da70bf1b731347ad73051863899990a89701b207e6d4d0f4fb791531 WatchSource:0}: Error finding container 53f46880da70bf1b731347ad73051863899990a89701b207e6d4d0f4fb791531: Status 404 returned error can't find the container with id 53f46880da70bf1b731347ad73051863899990a89701b207e6d4d0f4fb791531
Jul 28 14:24:10.419223 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:10.419139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:10.419370 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:10.419239    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 14:24:10.419370 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:10.419291    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls podName:1f8fec6e-7ff9-4c0c-b177-c7c5893549a4 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:11.419278402 +0000 UTC m=+89.105356493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls") pod "cluster-samples-operator-6748489656-9mvr7" (UID: "1f8fec6e-7ff9-4c0c-b177-c7c5893549a4") : secret "samples-operator-tls" not found
Jul 28 14:24:11.190297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:11.190256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-skv6z" event={"ID":"ddbd6182-8e34-4f55-9518-56d442d0c058","Type":"ContainerStarted","Data":"53f46880da70bf1b731347ad73051863899990a89701b207e6d4d0f4fb791531"}
Jul 28 14:24:11.225030 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:11.224995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:11.225192 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:11.225037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:11.225192 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:11.225148    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:11.225192 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:11.225184    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:13.225167227 +0000 UTC m=+90.911245318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:11.225300 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:11.225202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:13.225195157 +0000 UTC m=+90.911273247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:11.427000 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:11.426968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:11.427317 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:11.427115    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 14:24:11.427317 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:11.427181    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls podName:1f8fec6e-7ff9-4c0c-b177-c7c5893549a4 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:13.427163415 +0000 UTC m=+91.113241505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls") pod "cluster-samples-operator-6748489656-9mvr7" (UID: "1f8fec6e-7ff9-4c0c-b177-c7c5893549a4") : secret "samples-operator-tls" not found
Jul 28 14:24:13.199905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.199879    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/0.log"
Jul 28 14:24:13.200323 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.199920    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10" containerID="e48ffcbd6bb14baaafc63663bcfed585ffa05bdfb55eb8308e35573ab7253df0" exitCode=255
Jul 28 14:24:13.200323 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.199953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" event={"ID":"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10","Type":"ContainerDied","Data":"e48ffcbd6bb14baaafc63663bcfed585ffa05bdfb55eb8308e35573ab7253df0"}
Jul 28 14:24:13.200323 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.200260    2577 scope.go:117] "RemoveContainer" containerID="e48ffcbd6bb14baaafc63663bcfed585ffa05bdfb55eb8308e35573ab7253df0"
Jul 28 14:24:13.201543 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.201512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-skv6z" event={"ID":"ddbd6182-8e34-4f55-9518-56d442d0c058","Type":"ContainerStarted","Data":"2ea53a2cec214c51bdac93eca36358b5bfe2abe73d7b4133e6925c0bd8c0490e"}
Jul 28 14:24:13.242774 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.242744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:13.242920 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.242848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:13.242920 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:13.242901    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:13.243038 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:13.242971    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:17.242950856 +0000 UTC m=+94.929028948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:13.243391 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:13.243361    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:17.243344447 +0000 UTC m=+94.929422539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:13.444655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:13.444611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:13.444831 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:13.444755    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 14:24:13.444831 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:13.444819    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls podName:1f8fec6e-7ff9-4c0c-b177-c7c5893549a4 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:17.444803878 +0000 UTC m=+95.130881973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls") pod "cluster-samples-operator-6748489656-9mvr7" (UID: "1f8fec6e-7ff9-4c0c-b177-c7c5893549a4") : secret "samples-operator-tls" not found
Jul 28 14:24:14.206004 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.205976    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/1.log"
Jul 28 14:24:14.206401 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.206361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/0.log"
Jul 28 14:24:14.206401 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.206388    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34" exitCode=255
Jul 28 14:24:14.206473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.206421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" event={"ID":"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10","Type":"ContainerDied","Data":"4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"}
Jul 28 14:24:14.206507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.206492    2577 scope.go:117] "RemoveContainer" containerID="e48ffcbd6bb14baaafc63663bcfed585ffa05bdfb55eb8308e35573ab7253df0"
Jul 28 14:24:14.206698 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.206673    2577 scope.go:117] "RemoveContainer" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"
Jul 28 14:24:14.206882 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:14.206862    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:14.233952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.233900    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-skv6z" podStartSLOduration=2.449535112 podStartE2EDuration="5.23388444s" totalImagesPullingTime="2.784349328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:13.234804645 +0000 UTC m=+90.920882759" watchObservedRunningTime="2026-07-28 14:24:14.23388444 +0000 UTC m=+91.919962552"
Jul 28 14:24:14.997692 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:14.997659    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zsnpr"]
Jul 28 14:24:15.002590 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.002563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.004964 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.004945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 14:24:15.005414 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.005399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 14:24:15.005765 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.005747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wl4mt\""
Jul 28 14:24:15.013357 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.013333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zsnpr"]
Jul 28 14:24:15.057184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.057146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.057351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.057260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/103f1358-62b6-466c-a0a5-61007d6c4506-crio-socket\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.057351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.057292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/103f1358-62b6-466c-a0a5-61007d6c4506-data-volume\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.057351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.057325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.057351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.057344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k54n\" (UniqueName: \"kubernetes.io/projected/103f1358-62b6-466c-a0a5-61007d6c4506-kube-api-access-2k54n\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158103 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/103f1358-62b6-466c-a0a5-61007d6c4506-crio-socket\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158103 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/103f1358-62b6-466c-a0a5-61007d6c4506-data-volume\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k54n\" (UniqueName: \"kubernetes.io/projected/103f1358-62b6-466c-a0a5-61007d6c4506-kube-api-access-2k54n\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/103f1358-62b6-466c-a0a5-61007d6c4506-crio-socket\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.158225    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.158302 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.158300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:15.658284296 +0000 UTC m=+93.344362387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.158494 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.158335    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.158494 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.158396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:15.658379387 +0000 UTC m=+93.344457486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.158494 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.158455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/103f1358-62b6-466c-a0a5-61007d6c4506-data-volume\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.167024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.167002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k54n\" (UniqueName: \"kubernetes.io/projected/103f1358-62b6-466c-a0a5-61007d6c4506-kube-api-access-2k54n\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.211434 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.211411    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/1.log"
Jul 28 14:24:15.211888 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.211826    2577 scope.go:117] "RemoveContainer" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"
Jul 28 14:24:15.212027 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.212005    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:15.228688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.228665    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-46766"]
Jul 28 14:24:15.231469 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.231454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.233742 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.233720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 14:24:15.234258 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.234235    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bwtgs\""
Jul 28 14:24:15.234366 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.234311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 14:24:15.248006 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.247952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-46766"]
Jul 28 14:24:15.259292 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.259270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/626a50b6-d25d-45ee-925a-ac17fa5e8990-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.259358 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.259335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.336447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.336418    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"]
Jul 28 14:24:15.339344 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.339326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"
Jul 28 14:24:15.342106 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.342085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 14:24:15.342194 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.342149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mngqm\""
Jul 28 14:24:15.342237 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.342203    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 14:24:15.355443 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.355418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"]
Jul 28 14:24:15.360288 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.360268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/626a50b6-d25d-45ee-925a-ac17fa5e8990-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.360395 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.360326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.360455 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.360417    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 14:24:15.360507 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.360464    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert podName:626a50b6-d25d-45ee-925a-ac17fa5e8990 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:15.86044681 +0000 UTC m=+93.546524905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-46766" (UID: "626a50b6-d25d-45ee-925a-ac17fa5e8990") : secret "networking-console-plugin-cert" not found
Jul 28 14:24:15.360936 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.360915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/626a50b6-d25d-45ee-925a-ac17fa5e8990-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.461575 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.461515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7vc\" (UniqueName: \"kubernetes.io/projected/1e63d39f-f71a-4fdd-a996-ceb64b18ba68-kube-api-access-pf7vc\") pod \"migrator-6865c9967-v5w5j\" (UID: \"1e63d39f-f71a-4fdd-a996-ceb64b18ba68\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"
Jul 28 14:24:15.562925 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.562834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7vc\" (UniqueName: \"kubernetes.io/projected/1e63d39f-f71a-4fdd-a996-ceb64b18ba68-kube-api-access-pf7vc\") pod \"migrator-6865c9967-v5w5j\" (UID: \"1e63d39f-f71a-4fdd-a996-ceb64b18ba68\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"
Jul 28 14:24:15.578401 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.578372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7vc\" (UniqueName: \"kubernetes.io/projected/1e63d39f-f71a-4fdd-a996-ceb64b18ba68-kube-api-access-pf7vc\") pod \"migrator-6865c9967-v5w5j\" (UID: \"1e63d39f-f71a-4fdd-a996-ceb64b18ba68\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"
Jul 28 14:24:15.647922 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.647887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"
Jul 28 14:24:15.664282 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.664255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.664406 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.664315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:15.664406 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.664332    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.664406 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.664393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:16.664378837 +0000 UTC m=+94.350456947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.664566 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.664413    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.664566 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.664454    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:16.664443646 +0000 UTC m=+94.350521737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.768764 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.768739    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j"]
Jul 28 14:24:15.770951 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:15.770914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e63d39f_f71a_4fdd_a996_ceb64b18ba68.slice/crio-b43d830db09a08168b3d50a8ed64e5ccfd15c6871a71400da660cd779e2f1c4a WatchSource:0}: Error finding container b43d830db09a08168b3d50a8ed64e5ccfd15c6871a71400da660cd779e2f1c4a: Status 404 returned error can't find the container with id b43d830db09a08168b3d50a8ed64e5ccfd15c6871a71400da660cd779e2f1c4a
Jul 28 14:24:15.866042 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:15.865931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:15.866214 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.866093    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 14:24:15.866214 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:15.866156    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert podName:626a50b6-d25d-45ee-925a-ac17fa5e8990 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:16.866140541 +0000 UTC m=+94.552218632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-46766" (UID: "626a50b6-d25d-45ee-925a-ac17fa5e8990") : secret "networking-console-plugin-cert" not found
Jul 28 14:24:16.215111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:16.215075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j" event={"ID":"1e63d39f-f71a-4fdd-a996-ceb64b18ba68","Type":"ContainerStarted","Data":"b43d830db09a08168b3d50a8ed64e5ccfd15c6871a71400da660cd779e2f1c4a"}
Jul 28 14:24:16.674238 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:16.674155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:16.674238 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:16.674214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:16.674475 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.674287    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:16.674475 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.674327    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:16.674475 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.674366    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:18.674345834 +0000 UTC m=+96.360423940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:16.674475 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.674385    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:18.674376904 +0000 UTC m=+96.360454994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:16.875194 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:16.875150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:16.875379 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.875265    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 14:24:16.875379 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:16.875350    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert podName:626a50b6-d25d-45ee-925a-ac17fa5e8990 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:18.875329265 +0000 UTC m=+96.561407356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-46766" (UID: "626a50b6-d25d-45ee-925a-ac17fa5e8990") : secret "networking-console-plugin-cert" not found
Jul 28 14:24:16.972322 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:16.972297    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jntvr_d0c388cc-d3a8-4cee-aea7-8eadddae330e/dns-node-resolver/0.log"
Jul 28 14:24:17.219975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.219891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j" event={"ID":"1e63d39f-f71a-4fdd-a996-ceb64b18ba68","Type":"ContainerStarted","Data":"649d437c92d9aa234b3f3d549f2b9b4da0963c163fc2cdaee4aa5954f3929d62"}
Jul 28 14:24:17.219975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.219971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j" event={"ID":"1e63d39f-f71a-4fdd-a996-ceb64b18ba68","Type":"ContainerStarted","Data":"e8d181a821d98b1fdb1c0e3c546c64a993c7e48ed3321ff0eb8d2c66d958b145"}
Jul 28 14:24:17.237679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.237622    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-v5w5j" podStartSLOduration=1.038728924 podStartE2EDuration="2.237605186s" totalImagesPullingTime="1.198876262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:17.236109601 +0000 UTC m=+94.922187714" watchObservedRunningTime="2026-07-28 14:24:17.237605186 +0000 UTC m=+94.923683299"
Jul 28 14:24:17.279366 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.279331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:17.279526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.279379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:17.279526 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:17.279494    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:25.279476938 +0000 UTC m=+102.965555029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:17.279643 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:17.279601    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:17.279681 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:17.279656    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:25.279644103 +0000 UTC m=+102.965722194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:17.481394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.481291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:17.481580 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:17.481441    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 14:24:17.481580 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:17.481511    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls podName:1f8fec6e-7ff9-4c0c-b177-c7c5893549a4 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:25.481494493 +0000 UTC m=+103.167572586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls") pod "cluster-samples-operator-6748489656-9mvr7" (UID: "1f8fec6e-7ff9-4c0c-b177-c7c5893549a4") : secret "samples-operator-tls" not found
Jul 28 14:24:17.772016 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:17.771946    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckjp9_a9641f09-cad8-4bde-9109-1a7da203da71/node-ca/0.log"
Jul 28 14:24:18.020252 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.020219    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jc7gq"]
Jul 28 14:24:18.023319 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.023278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.025809 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.025772    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 14:24:18.025960 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.025884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 14:24:18.025960 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.025931    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 14:24:18.026712 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.026695    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 14:24:18.026800 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.026729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jsz6s\""
Jul 28 14:24:18.032126 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.032107    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jc7gq"]
Jul 28 14:24:18.086667 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.086628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-key\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.086667 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.086672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.086867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.086739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfss4\" (UniqueName: \"kubernetes.io/projected/368f70e4-e1fa-4723-bf75-22fdd9effaa5-kube-api-access-bfss4\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.187447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.187410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-key\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.187447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.187449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.187677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.187485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfss4\" (UniqueName: \"kubernetes.io/projected/368f70e4-e1fa-4723-bf75-22fdd9effaa5-kube-api-access-bfss4\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.188200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.188179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-cabundle\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.189809 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.189792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/368f70e4-e1fa-4723-bf75-22fdd9effaa5-signing-key\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.195994 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.195969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfss4\" (UniqueName: \"kubernetes.io/projected/368f70e4-e1fa-4723-bf75-22fdd9effaa5-kube-api-access-bfss4\") pod \"service-ca-658d7b7c78-jc7gq\" (UID: \"368f70e4-e1fa-4723-bf75-22fdd9effaa5\") " pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.332420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.332335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq"
Jul 28 14:24:18.454371 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.454342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-jc7gq"]
Jul 28 14:24:18.458157 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:18.458129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368f70e4_e1fa_4723_bf75_22fdd9effaa5.slice/crio-92c6f02957f7dc19f97c19d0fdcde9718541f18620e9dfdea4ae8839629c58e9 WatchSource:0}: Error finding container 92c6f02957f7dc19f97c19d0fdcde9718541f18620e9dfdea4ae8839629c58e9: Status 404 returned error can't find the container with id 92c6f02957f7dc19f97c19d0fdcde9718541f18620e9dfdea4ae8839629c58e9
Jul 28 14:24:18.691706 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.691622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:18.691866 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.691742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:18.691866 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.691776    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:18.691866 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.691806    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:18.691866 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.691841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:22.691825237 +0000 UTC m=+100.377903328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:18.691866 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.691855    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:22.69184937 +0000 UTC m=+100.377927461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:18.893506 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.893473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:18.893664 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.893638    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 14:24:18.893715 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:18.893705    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert podName:626a50b6-d25d-45ee-925a-ac17fa5e8990 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:22.893689461 +0000 UTC m=+100.579767552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-46766" (UID: "626a50b6-d25d-45ee-925a-ac17fa5e8990") : secret "networking-console-plugin-cert" not found
Jul 28 14:24:18.972660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:18.972600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7z6c_958f61c6-5bc6-42e7-ac00-c6e86ec3d35c/serve-healthcheck-canary/0.log"
Jul 28 14:24:19.227580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:19.227480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq" event={"ID":"368f70e4-e1fa-4723-bf75-22fdd9effaa5","Type":"ContainerStarted","Data":"92c6f02957f7dc19f97c19d0fdcde9718541f18620e9dfdea4ae8839629c58e9"}
Jul 28 14:24:19.988372 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:19.988332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:19.988824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:19.988407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:19.988824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:19.988795    2577 scope.go:117] "RemoveContainer" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"
Jul 28 14:24:19.989002 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:19.988973    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:21.235093 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:21.235052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq" event={"ID":"368f70e4-e1fa-4723-bf75-22fdd9effaa5","Type":"ContainerStarted","Data":"ba210ae319ea5b18cdbbc07b1aba757e9040177a54cbc4fc8204a581ec8f1ae7"}
Jul 28 14:24:21.252800 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:21.252754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-jc7gq" podStartSLOduration=1.289680212 podStartE2EDuration="3.252738616s" totalImagesPullingTime="1.963058404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:21.25162551 +0000 UTC m=+98.937703624" watchObservedRunningTime="2026-07-28 14:24:21.252738616 +0000 UTC m=+98.938816728"
Jul 28 14:24:22.024650 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:22.024469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:24:22.025350 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.024622    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:24:22.025608 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.025586    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls podName:3529d4aa-3f68-4339-824d-dbfc4872c9ce nodeName:}" failed. No retries permitted until 2026-07-28 14:25:26.025547714 +0000 UTC m=+163.711625821 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls") pod "dns-default-9chw8" (UID: "3529d4aa-3f68-4339-824d-dbfc4872c9ce") : secret "dns-default-metrics-tls" not found
Jul 28 14:24:22.730751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:22.730711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:22.731143 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:22.730785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:22.731143 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.730830    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:22.731143 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.730910    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:30.730888659 +0000 UTC m=+108.416966767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:22.731143 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.730932    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:22.731143 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.730994    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:30.730981661 +0000 UTC m=+108.417059765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:22.932401 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:22.932371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:22.932592 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.932529    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 14:24:22.932636 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:22.932613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert podName:626a50b6-d25d-45ee-925a-ac17fa5e8990 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:30.932596554 +0000 UTC m=+108.618674653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-46766" (UID: "626a50b6-d25d-45ee-925a-ac17fa5e8990") : secret "networking-console-plugin-cert" not found
Jul 28 14:24:25.149261 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.149230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x2n29"
Jul 28 14:24:25.355368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.355321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:25.355368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.355370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:25.355659 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:25.355467    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 14:24:25.355659 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:25.355562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:41.35552368 +0000 UTC m=+119.041601771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : secret "router-metrics-certs-default" not found
Jul 28 14:24:25.355659 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:25.355608    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle podName:3300c232-216d-424b-9ce2-7220f6dbf8a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:41.355589008 +0000 UTC m=+119.041667104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle") pod "router-default-757957ffc-frc2z" (UID: "3300c232-216d-424b-9ce2-7220f6dbf8a3") : configmap references non-existent config key: service-ca.crt
Jul 28 14:24:25.557368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.557328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:25.559679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.559657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f8fec6e-7ff9-4c0c-b177-c7c5893549a4-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9mvr7\" (UID: \"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:25.685341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.685302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"
Jul 28 14:24:25.838992 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:25.838924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7"]
Jul 28 14:24:26.255635 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:26.255600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7" event={"ID":"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4","Type":"ContainerStarted","Data":"dea538fce59b80d3ac9e40753207c97228b1f9299a2c03336664ade58e49b7ff"}
Jul 28 14:24:28.264418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:28.264333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7" event={"ID":"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4","Type":"ContainerStarted","Data":"1f7ad2474aa6d4f8be5ebb02c2a5a98dd44fc8bf1367e8c2d95fd6bb43b6c64e"}
Jul 28 14:24:28.264418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:28.264405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7" event={"ID":"1f8fec6e-7ff9-4c0c-b177-c7c5893549a4","Type":"ContainerStarted","Data":"523a60b34eebbd4907a56e895a00cee871f9b86b8b280f946303e233cc7e3992"}
Jul 28 14:24:28.286894 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:28.286850    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9mvr7" podStartSLOduration=17.184830274 podStartE2EDuration="19.286835174s" totalImagesPullingTime="2.1020049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:28.286813593 +0000 UTC m=+105.972891707" watchObservedRunningTime="2026-07-28 14:24:28.286835174 +0000 UTC m=+105.972913288"
Jul 28 14:24:30.803984 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:30.803941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:30.804470 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:30.804010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:30.804470 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:30.804147    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:30.804470 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:30.804230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls podName:103f1358-62b6-466c-a0a5-61007d6c4506 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:46.804207501 +0000 UTC m=+124.490285595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zsnpr" (UID: "103f1358-62b6-466c-a0a5-61007d6c4506") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:30.804669 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:30.804567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/103f1358-62b6-466c-a0a5-61007d6c4506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:31.005749 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:31.005698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:31.008105 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:31.008067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/626a50b6-d25d-45ee-925a-ac17fa5e8990-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-46766\" (UID: \"626a50b6-d25d-45ee-925a-ac17fa5e8990\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:31.140327 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:31.140237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766"
Jul 28 14:24:31.264205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:31.264170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-46766"]
Jul 28 14:24:31.268287 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:31.268251    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626a50b6_d25d_45ee_925a_ac17fa5e8990.slice/crio-bed9d8a254dc9c14f7de639fe58e5e26e4eff8b423a8de77566f9f631ca56fa7 WatchSource:0}: Error finding container bed9d8a254dc9c14f7de639fe58e5e26e4eff8b423a8de77566f9f631ca56fa7: Status 404 returned error can't find the container with id bed9d8a254dc9c14f7de639fe58e5e26e4eff8b423a8de77566f9f631ca56fa7
Jul 28 14:24:31.274458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:31.274436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766" event={"ID":"626a50b6-d25d-45ee-925a-ac17fa5e8990","Type":"ContainerStarted","Data":"bed9d8a254dc9c14f7de639fe58e5e26e4eff8b423a8de77566f9f631ca56fa7"}
Jul 28 14:24:32.781009 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:32.780919    2577 scope.go:117] "RemoveContainer" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"
Jul 28 14:24:33.282382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.282355    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:24:33.282741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.282724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/1.log"
Jul 28 14:24:33.282818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.282758    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10" containerID="431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d" exitCode=255
Jul 28 14:24:33.282861 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.282831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" event={"ID":"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10","Type":"ContainerDied","Data":"431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d"}
Jul 28 14:24:33.282910 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.282897    2577 scope.go:117] "RemoveContainer" containerID="4ab6726e037635c575f0f5beb4794b3663a2aa846fa56166b571051bcc0f6b34"
Jul 28 14:24:33.283216 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.283194    2577 scope.go:117] "RemoveContainer" containerID="431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d"
Jul 28 14:24:33.283400 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:33.283379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:33.284356 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.284340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766" event={"ID":"626a50b6-d25d-45ee-925a-ac17fa5e8990","Type":"ContainerStarted","Data":"83cbe453642c17a519344330f83a0687f9ec0cfdca2c1ff1e0625fca208ec512"}
Jul 28 14:24:33.316765 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:33.316715    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-46766" podStartSLOduration=17.061588812 podStartE2EDuration="18.316696113s" totalImagesPullingTime="1.255107301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:33.315408484 +0000 UTC m=+111.001486597" watchObservedRunningTime="2026-07-28 14:24:33.316696113 +0000 UTC m=+111.002774222"
Jul 28 14:24:34.289313 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:34.289288    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:24:39.988313 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:39.988277    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:39.988816 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:39.988344    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:24:39.988816 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:39.988766    2577 scope.go:117] "RemoveContainer" containerID="431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d"
Jul 28 14:24:39.988985 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:39.988961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:41.390309 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.390248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:41.390840 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.390398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:41.390929 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.390908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3300c232-216d-424b-9ce2-7220f6dbf8a3-service-ca-bundle\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:41.392740 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.392720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3300c232-216d-424b-9ce2-7220f6dbf8a3-metrics-certs\") pod \"router-default-757957ffc-frc2z\" (UID: \"3300c232-216d-424b-9ce2-7220f6dbf8a3\") " pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:41.673754 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.673660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:41.832020 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:41.831987    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-757957ffc-frc2z"]
Jul 28 14:24:41.836127 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:41.836096    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3300c232_216d_424b_9ce2_7220f6dbf8a3.slice/crio-4c27fc284577731f151df0b67f4283603bc89f27e4a71cc32f1177b89722fe52 WatchSource:0}: Error finding container 4c27fc284577731f151df0b67f4283603bc89f27e4a71cc32f1177b89722fe52: Status 404 returned error can't find the container with id 4c27fc284577731f151df0b67f4283603bc89f27e4a71cc32f1177b89722fe52
Jul 28 14:24:42.319642 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:42.319608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-757957ffc-frc2z" event={"ID":"3300c232-216d-424b-9ce2-7220f6dbf8a3","Type":"ContainerStarted","Data":"2da844b4daa5721078ecf0969e1fc3c90c326c727fd0379e673847f82899c5e5"}
Jul 28 14:24:42.319830 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:42.319656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-757957ffc-frc2z" event={"ID":"3300c232-216d-424b-9ce2-7220f6dbf8a3","Type":"ContainerStarted","Data":"4c27fc284577731f151df0b67f4283603bc89f27e4a71cc32f1177b89722fe52"}
Jul 28 14:24:42.341921 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:42.341868    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-757957ffc-frc2z" podStartSLOduration=33.341853391 podStartE2EDuration="33.341853391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:42.340399285 +0000 UTC m=+120.026477397" watchObservedRunningTime="2026-07-28 14:24:42.341853391 +0000 UTC m=+120.027931510"
Jul 28 14:24:42.674172 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:42.674080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:42.676526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:42.676503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:43.322845 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:43.322808    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:43.324082 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:43.324064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-757957ffc-frc2z"
Jul 28 14:24:46.834002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:46.833965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:46.836217 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:46.836181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/103f1358-62b6-466c-a0a5-61007d6c4506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zsnpr\" (UID: \"103f1358-62b6-466c-a0a5-61007d6c4506\") " pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:47.116211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.116136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wl4mt\""
Jul 28 14:24:47.121201 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.121184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zsnpr"
Jul 28 14:24:47.246269 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.246236    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zsnpr"]
Jul 28 14:24:47.249146 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:47.249117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103f1358_62b6_466c_a0a5_61007d6c4506.slice/crio-b0b4772df63748e0a8323849d5c87fc60657584b6f219b08c69d67d9a1821e0d WatchSource:0}: Error finding container b0b4772df63748e0a8323849d5c87fc60657584b6f219b08c69d67d9a1821e0d: Status 404 returned error can't find the container with id b0b4772df63748e0a8323849d5c87fc60657584b6f219b08c69d67d9a1821e0d
Jul 28 14:24:47.337044 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.337010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zsnpr" event={"ID":"103f1358-62b6-466c-a0a5-61007d6c4506","Type":"ContainerStarted","Data":"b0b4772df63748e0a8323849d5c87fc60657584b6f219b08c69d67d9a1821e0d"}
Jul 28 14:24:47.843557 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.843512    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-88s9g"]
Jul 28 14:24:47.845873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.845847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.850652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.850870    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.851047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.851232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xzfsp\""
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.851451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 14:24:47.852295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.851976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 14:24:47.857066 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.856768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-88s9g"]
Jul 28 14:24:47.945111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.944798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtp49\" (UniqueName: \"kubernetes.io/projected/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-kube-api-access-rtp49\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:47.945111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.944911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:47.945111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.944945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:47.945111 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:47.944972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.045479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtp49\" (UniqueName: \"kubernetes.io/projected/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-kube-api-access-rtp49\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.045594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.045627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.045654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:48.046291    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:48.046366    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls podName:6bb46449-8486-46b3-9ab2-5c1b3f52ad38 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:48.546345856 +0000 UTC m=+126.232423952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-88s9g" (UID: "6bb46449-8486-46b3-9ab2-5c1b3f52ad38") : secret "prometheus-operator-tls" not found
Jul 28 14:24:48.046567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.046504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.049602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.049524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.055100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.055073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtp49\" (UniqueName: \"kubernetes.io/projected/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-kube-api-access-rtp49\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.550840 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.550799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.553474 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.553450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bb46449-8486-46b3-9ab2-5c1b3f52ad38-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-88s9g\" (UID: \"6bb46449-8486-46b3-9ab2-5c1b3f52ad38\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.768775 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.768740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g"
Jul 28 14:24:48.928551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:48.928499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-88s9g"]
Jul 28 14:24:48.931909 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:48.931880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb46449_8486_46b3_9ab2_5c1b3f52ad38.slice/crio-14e965db5c6a17c78d8f59db7ac13ef7a7d898fe584c211a8bc949929fdf2ff6 WatchSource:0}: Error finding container 14e965db5c6a17c78d8f59db7ac13ef7a7d898fe584c211a8bc949929fdf2ff6: Status 404 returned error can't find the container with id 14e965db5c6a17c78d8f59db7ac13ef7a7d898fe584c211a8bc949929fdf2ff6
Jul 28 14:24:49.345337 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:49.345296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g" event={"ID":"6bb46449-8486-46b3-9ab2-5c1b3f52ad38","Type":"ContainerStarted","Data":"14e965db5c6a17c78d8f59db7ac13ef7a7d898fe584c211a8bc949929fdf2ff6"}
Jul 28 14:24:50.780153 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:50.780125    2577 scope.go:117] "RemoveContainer" containerID="431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d"
Jul 28 14:24:50.780593 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:24:50.780314    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-2mw5m_openshift-console-operator(ef95864b-dd12-48f1-b4f7-69eb0e2d0f10)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podUID="ef95864b-dd12-48f1-b4f7-69eb0e2d0f10"
Jul 28 14:24:52.585064 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:52.585037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:24:52.587249 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:52.587225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b31b90ef-b78e-40a5-89c7-b36dabbc993d-metrics-certs\") pod \"network-metrics-daemon-sqcgv\" (UID: \"b31b90ef-b78e-40a5-89c7-b36dabbc993d\") " pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:24:52.800288 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:52.800217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z6ktw\""
Jul 28 14:24:52.808459 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:52.808439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqcgv"
Jul 28 14:24:52.931688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:52.931661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqcgv"]
Jul 28 14:24:52.934855 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:52.934830    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31b90ef_b78e_40a5_89c7_b36dabbc993d.slice/crio-81473be73fc01644c92f85b6e56e1a35bf200c8ca3e3be7f4d78f30871ba59c5 WatchSource:0}: Error finding container 81473be73fc01644c92f85b6e56e1a35bf200c8ca3e3be7f4d78f30871ba59c5: Status 404 returned error can't find the container with id 81473be73fc01644c92f85b6e56e1a35bf200c8ca3e3be7f4d78f30871ba59c5
Jul 28 14:24:53.363027 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:53.362482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g" event={"ID":"6bb46449-8486-46b3-9ab2-5c1b3f52ad38","Type":"ContainerStarted","Data":"1d9eee7261ebe4b023d0aa56ec18f0ab3f51d37491a4514d2a76d1ac4c028204"}
Jul 28 14:24:53.363027 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:53.362564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g" event={"ID":"6bb46449-8486-46b3-9ab2-5c1b3f52ad38","Type":"ContainerStarted","Data":"dfb70e085963d5ba150e047fc6fa19bd45ba464cb2464ee35937747d5ceab963"}
Jul 28 14:24:53.365181 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:53.365145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zsnpr" event={"ID":"103f1358-62b6-466c-a0a5-61007d6c4506","Type":"ContainerStarted","Data":"6e23fa3b25208e1140e728261e74c0afe47419e1427e691cd462e3a109d9c01d"}
Jul 28 14:24:53.367228 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:53.367170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqcgv" event={"ID":"b31b90ef-b78e-40a5-89c7-b36dabbc993d","Type":"ContainerStarted","Data":"81473be73fc01644c92f85b6e56e1a35bf200c8ca3e3be7f4d78f30871ba59c5"}
Jul 28 14:24:53.382426 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:53.382216    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-88s9g" podStartSLOduration=2.818231362 podStartE2EDuration="6.382200427s" totalImagesPullingTime="3.563969065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:53.381476689 +0000 UTC m=+131.067554805" watchObservedRunningTime="2026-07-28 14:24:53.382200427 +0000 UTC m=+131.068278541"
Jul 28 14:24:54.373382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:54.373289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqcgv" event={"ID":"b31b90ef-b78e-40a5-89c7-b36dabbc993d","Type":"ContainerStarted","Data":"ce801e53c9e80a04c1dbd4f2d484a2816e4dc5788613c06370f06b1edb3c9d74"}
Jul 28 14:24:54.373382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:54.373354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqcgv" event={"ID":"b31b90ef-b78e-40a5-89c7-b36dabbc993d","Type":"ContainerStarted","Data":"e51073cc3cf851f7691d4d16f6b0e33a3d13718a9b729d00553cfb3f1d8721fe"}
Jul 28 14:24:54.391265 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:54.391213    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sqcgv" podStartSLOduration=130.338879604 podStartE2EDuration="2m11.391196655s" totalImagesPullingTime="1.052317051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:54.390744151 +0000 UTC m=+132.076822264" watchObservedRunningTime="2026-07-28 14:24:54.391196655 +0000 UTC m=+132.077274767"
Jul 28 14:24:55.285405 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.285362    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"]
Jul 28 14:24:55.307126 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.306314    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r5vxj"]
Jul 28 14:24:55.307126 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.306530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.310465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.309638    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-swmzk\""
Jul 28 14:24:55.310465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.309784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 14:24:55.310465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.309817    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 14:24:55.310465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.310352    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 14:24:55.312420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.312372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"]
Jul 28 14:24:55.312586 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.312506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.316840 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.316818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 14:24:55.317130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.316851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5brqd\""
Jul 28 14:24:55.317272 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.316936    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 14:24:55.317769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.317644    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 14:24:55.407397 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-tls\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-root\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-sys\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88268\" (UniqueName: \"kubernetes.io/projected/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-api-access-88268\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-wtmp\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x25x\" (UniqueName: \"kubernetes.io/projected/3eee1f1f-681b-41d3-8d3f-0b9990289f97-kube-api-access-9x25x\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af3ef47-00b0-4639-9d94-f99cf0334097-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-textfile\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.407864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.408393 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.408393 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.407959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-metrics-client-ca\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.408393 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.408003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.508837 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.508805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509030 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.508847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-metrics-client-ca\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509030 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.508878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509030 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.508996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-tls\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-root\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-sys\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88268\" (UniqueName: \"kubernetes.io/projected/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-api-access-88268\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-wtmp\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x25x\" (UniqueName: \"kubernetes.io/projected/3eee1f1f-681b-41d3-8d3f-0b9990289f97-kube-api-access-9x25x\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af3ef47-00b0-4639-9d94-f99cf0334097-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-textfile\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.509489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-metrics-client-ca\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-wtmp\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.509769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.510293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.509952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-sys\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.510293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.510069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eee1f1f-681b-41d3-8d3f-0b9990289f97-root\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.510293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.510189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.510501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.510340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1af3ef47-00b0-4639-9d94-f99cf0334097-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.510501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.510396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-textfile\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.510501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.510442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1af3ef47-00b0-4639-9d94-f99cf0334097-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.512389 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.512367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.512554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.512435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.512554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.512451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.512791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.512772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eee1f1f-681b-41d3-8d3f-0b9990289f97-node-exporter-tls\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.523164 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.523121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88268\" (UniqueName: \"kubernetes.io/projected/1af3ef47-00b0-4639-9d94-f99cf0334097-kube-api-access-88268\") pod \"kube-state-metrics-56d9f944b4-jnzrh\" (UID: \"1af3ef47-00b0-4639-9d94-f99cf0334097\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.524391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.524369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x25x\" (UniqueName: \"kubernetes.io/projected/3eee1f1f-681b-41d3-8d3f-0b9990289f97-kube-api-access-9x25x\") pod \"node-exporter-r5vxj\" (UID: \"3eee1f1f-681b-41d3-8d3f-0b9990289f97\") " pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:55.621663 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.621596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"
Jul 28 14:24:55.628906 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:55.628885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r5vxj"
Jul 28 14:24:56.258245 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:56.258210    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eee1f1f_681b_41d3_8d3f_0b9990289f97.slice/crio-cd0ea9570b43e35ec2ffcdbf4910f5d244d9bcd0ad22f12b68b74da355662a72 WatchSource:0}: Error finding container cd0ea9570b43e35ec2ffcdbf4910f5d244d9bcd0ad22f12b68b74da355662a72: Status 404 returned error can't find the container with id cd0ea9570b43e35ec2ffcdbf4910f5d244d9bcd0ad22f12b68b74da355662a72
Jul 28 14:24:56.386349 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:56.386318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5vxj" event={"ID":"3eee1f1f-681b-41d3-8d3f-0b9990289f97","Type":"ContainerStarted","Data":"cd0ea9570b43e35ec2ffcdbf4910f5d244d9bcd0ad22f12b68b74da355662a72"}
Jul 28 14:24:56.388426 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:56.388397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zsnpr" event={"ID":"103f1358-62b6-466c-a0a5-61007d6c4506","Type":"ContainerStarted","Data":"36fd6fb39e67d7cfa899a2d63e61360efa0152a93974c9ae5d7a3b84517c8898"}
Jul 28 14:24:56.401814 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:24:56.401783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af3ef47_00b0_4639_9d94_f99cf0334097.slice/crio-508abe19f61e4aaad8a370959c8780658cad317f96722ee6221826c8711ca687 WatchSource:0}: Error finding container 508abe19f61e4aaad8a370959c8780658cad317f96722ee6221826c8711ca687: Status 404 returned error can't find the container with id 508abe19f61e4aaad8a370959c8780658cad317f96722ee6221826c8711ca687
Jul 28 14:24:56.404685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:56.404667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh"]
Jul 28 14:24:57.394740 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:57.394709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh" event={"ID":"1af3ef47-00b0-4639-9d94-f99cf0334097","Type":"ContainerStarted","Data":"508abe19f61e4aaad8a370959c8780658cad317f96722ee6221826c8711ca687"}
Jul 28 14:24:58.400211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:58.400180    2577 generic.go:356] "Generic (PLEG): container finished" podID="3eee1f1f-681b-41d3-8d3f-0b9990289f97" containerID="4c2035fe61649460239f601ab12daba3c9d9528534c33c1bcf7d9ac4e7e3b662" exitCode=0
Jul 28 14:24:58.400679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:24:58.400242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5vxj" event={"ID":"3eee1f1f-681b-41d3-8d3f-0b9990289f97","Type":"ContainerDied","Data":"4c2035fe61649460239f601ab12daba3c9d9528534c33c1bcf7d9ac4e7e3b662"}
Jul 28 14:25:00.420075 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.419936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh" event={"ID":"1af3ef47-00b0-4639-9d94-f99cf0334097","Type":"ContainerStarted","Data":"9ffb1377711a4d168f1f2f3343d3882ada516c4edd47b3f8ee2d01b8971515a3"}
Jul 28 14:25:00.420075 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.420031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh" event={"ID":"1af3ef47-00b0-4639-9d94-f99cf0334097","Type":"ContainerStarted","Data":"c8c43ac8c6478061634fe28f9ccd444469a3b6eb653a38c208aebde5250f197f"}
Jul 28 14:25:00.423955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.423527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5vxj" event={"ID":"3eee1f1f-681b-41d3-8d3f-0b9990289f97","Type":"ContainerStarted","Data":"a6980ddc4a3e3e3d739c875d24d325d137b47b7371d0499f66ceaf3aaacd08f6"}
Jul 28 14:25:00.423955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.423608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r5vxj" event={"ID":"3eee1f1f-681b-41d3-8d3f-0b9990289f97","Type":"ContainerStarted","Data":"4549c505b73186725691fa2af59b14384abb131792cc318291f637b8497024ee"}
Jul 28 14:25:00.426379 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.426355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zsnpr" event={"ID":"103f1358-62b6-466c-a0a5-61007d6c4506","Type":"ContainerStarted","Data":"e904f7e5ac6beca5bd6e2feebfa198cb2a5807ca0bf10bba98692456ae3af679"}
Jul 28 14:25:00.447660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.447594    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r5vxj" podStartSLOduration=4.247243142 podStartE2EDuration="5.447578177s" totalImagesPullingTime="1.123971599s" totalInitContainerRuntime="76.363436ms" isStatefulPod=true podCreationTimestamp="2026-07-28 14:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:00.446479594 +0000 UTC m=+138.132557709" watchObservedRunningTime="2026-07-28 14:25:00.447578177 +0000 UTC m=+138.133656290"
Jul 28 14:25:00.468312 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:00.468258    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zsnpr" podStartSLOduration=33.633704901 podStartE2EDuration="46.468237604s" totalImagesPullingTime="12.834532703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:24:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:00.465515049 +0000 UTC m=+138.151593162" watchObservedRunningTime="2026-07-28 14:25:00.468237604 +0000 UTC m=+138.154315717"
Jul 28 14:25:01.432812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.432770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh" event={"ID":"1af3ef47-00b0-4639-9d94-f99cf0334097","Type":"ContainerStarted","Data":"5004c2d8a3334c57800dc5a9173b8fa0bb8466f024c295441d740e10a9e30f24"}
Jul 28 14:25:01.461030 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.460936    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jnzrh" podStartSLOduration=2.6507234889999998 podStartE2EDuration="6.460899431s" totalImagesPullingTime="3.810175942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:01.460034305 +0000 UTC m=+139.146112419" watchObservedRunningTime="2026-07-28 14:25:01.460899431 +0000 UTC m=+139.146977546"
Jul 28 14:25:01.591890 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.591853    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:25:01.596235 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.596206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.600100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 14:25:01.600383 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 14:25:01.600799 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600779    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 14:25:01.600895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600803    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 14:25:01.600895 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600804    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 14:25:01.601007 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 14:25:01.601007 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.600921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 14:25:01.601086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.601056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 14:25:01.601280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.601262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 14:25:01.601390 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.601374    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 14:25:01.601974 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.601951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9tbc\""
Jul 28 14:25:01.602131 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.602038    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5qr0i9a34brlr\""
Jul 28 14:25:01.602182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.602135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 14:25:01.602384 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.602369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 14:25:01.602518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.602470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 14:25:01.604183 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.604160    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 14:25:01.609790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.609770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 14:25:01.634336 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.634296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:25:01.665452 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665651 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.665842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvp5l\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.666095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.665949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp5l\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.766870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.766845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.767706 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.767678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.767867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.767726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768071 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768194 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768301 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768572 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768696 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768819 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.768933 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.768918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.769092 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.769021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.769856 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.769669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.770054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.770396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.770425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.770625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.771000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.771752 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.771677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.772896 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.771971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.772896 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.772394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.772896 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.772425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.772896 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.772550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.772896 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.772852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.773260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.773235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.773433 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.773413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.775024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.774998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.775125 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.775028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.775125 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.775116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.776432 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.776410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvp5l\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l\") pod \"prometheus-k8s-0\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:01.780600 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.780582    2577 scope.go:117] "RemoveContainer" containerID="431681d9269b0ac481e6350bcb180b65df9d32b02ae86f42d3299565b8592d9d"
Jul 28 14:25:01.909063 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:01.909024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:02.083755 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.083680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:25:02.085898 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:25:02.085868    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9147bc2_a56e_4ea7_8b8f_473762bf3fe9.slice/crio-0adc6a8f01800f3b9e87fa24904aef78c544b01aa03e0240c4775cf89d726182 WatchSource:0}: Error finding container 0adc6a8f01800f3b9e87fa24904aef78c544b01aa03e0240c4775cf89d726182: Status 404 returned error can't find the container with id 0adc6a8f01800f3b9e87fa24904aef78c544b01aa03e0240c4775cf89d726182
Jul 28 14:25:02.438836 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.438744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"0adc6a8f01800f3b9e87fa24904aef78c544b01aa03e0240c4775cf89d726182"}
Jul 28 14:25:02.440795 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.440772    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:25:02.440921 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.440898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" event={"ID":"ef95864b-dd12-48f1-b4f7-69eb0e2d0f10","Type":"ContainerStarted","Data":"934bae67c7dd662c377f56ed8103a4f9d825f71ff37e9d14e2d52c869047f0c9"}
Jul 28 14:25:02.441610 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.441562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:25:02.460676 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.460070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m" podStartSLOduration=50.574229723 podStartE2EDuration="53.460048311s" totalImagesPullingTime="2.885818588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:02.459631037 +0000 UTC m=+140.145709151" watchObservedRunningTime="2026-07-28 14:25:02.460048311 +0000 UTC m=+140.146126427"
Jul 28 14:25:02.476851 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:02.476601    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-2mw5m"
Jul 28 14:25:03.445701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:03.445668    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf" exitCode=0
Jul 28 14:25:03.446086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:03.445752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf"}
Jul 28 14:25:07.464411 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:07.464365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"}
Jul 28 14:25:07.464822 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:07.464458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"}
Jul 28 14:25:09.474790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:09.474759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"}
Jul 28 14:25:09.475175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:09.474820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"}
Jul 28 14:25:09.475175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:09.474848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"}
Jul 28 14:25:09.475175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:09.474871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerStarted","Data":"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"}
Jul 28 14:25:09.505390 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:09.505336    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.257716501 podStartE2EDuration="8.5053174s" totalImagesPullingTime="6.227047252s" totalInitContainerRuntime="20.553647ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:25:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:09.502295644 +0000 UTC m=+147.188373757" watchObservedRunningTime="2026-07-28 14:25:09.5053174 +0000 UTC m=+147.191395516"
Jul 28 14:25:11.909490 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:11.909452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:25:21.110298 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:25:21.110254    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9chw8" podUID="3529d4aa-3f68-4339-824d-dbfc4872c9ce"
Jul 28 14:25:21.524261 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:21.524232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:26.086607 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.086561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:26.088886 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.088864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3529d4aa-3f68-4339-824d-dbfc4872c9ce-metrics-tls\") pod \"dns-default-9chw8\" (UID: \"3529d4aa-3f68-4339-824d-dbfc4872c9ce\") " pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:26.327888 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.327858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl2dt\""
Jul 28 14:25:26.335968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.335946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:26.484750 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.484683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9chw8"]
Jul 28 14:25:26.487605 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:25:26.487572    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529d4aa_3f68_4339_824d_dbfc4872c9ce.slice/crio-65111c90b5d85c5db925fbb0839fdff74895fcfcf78a94647710ea2c552acf98 WatchSource:0}: Error finding container 65111c90b5d85c5db925fbb0839fdff74895fcfcf78a94647710ea2c552acf98: Status 404 returned error can't find the container with id 65111c90b5d85c5db925fbb0839fdff74895fcfcf78a94647710ea2c552acf98
Jul 28 14:25:26.545992 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:26.545957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9chw8" event={"ID":"3529d4aa-3f68-4339-824d-dbfc4872c9ce","Type":"ContainerStarted","Data":"65111c90b5d85c5db925fbb0839fdff74895fcfcf78a94647710ea2c552acf98"}
Jul 28 14:25:28.555708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:28.555670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9chw8" event={"ID":"3529d4aa-3f68-4339-824d-dbfc4872c9ce","Type":"ContainerStarted","Data":"2585c15e068bd5182d656572f333e7905fe3deb2083f7b449e9e4ab0b7f658de"}
Jul 28 14:25:28.556183 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:28.555724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9chw8" event={"ID":"3529d4aa-3f68-4339-824d-dbfc4872c9ce","Type":"ContainerStarted","Data":"f2cee1f5dee154845508b5b6210ede3d2070a84b37fa9433bdfb89c7e98e0549"}
Jul 28 14:25:28.556183 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:28.555805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:28.573616 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:28.573571    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9chw8" podStartSLOduration=129.123240514 podStartE2EDuration="2m10.573553384s" totalImagesPullingTime="1.45031287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:28.5722324 +0000 UTC m=+166.258310513" watchObservedRunningTime="2026-07-28 14:25:28.573553384 +0000 UTC m=+166.259631488"
Jul 28 14:25:38.562443 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:38.562362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9chw8"
Jul 28 14:25:44.619303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:44.619270    2577 generic.go:356] "Generic (PLEG): container finished" podID="ddbd6182-8e34-4f55-9518-56d442d0c058" containerID="2ea53a2cec214c51bdac93eca36358b5bfe2abe73d7b4133e6925c0bd8c0490e" exitCode=0
Jul 28 14:25:44.619769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:44.619319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-skv6z" event={"ID":"ddbd6182-8e34-4f55-9518-56d442d0c058","Type":"ContainerDied","Data":"2ea53a2cec214c51bdac93eca36358b5bfe2abe73d7b4133e6925c0bd8c0490e"}
Jul 28 14:25:44.619769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:44.619668    2577 scope.go:117] "RemoveContainer" containerID="2ea53a2cec214c51bdac93eca36358b5bfe2abe73d7b4133e6925c0bd8c0490e"
Jul 28 14:25:45.625325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:25:45.625284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-skv6z" event={"ID":"ddbd6182-8e34-4f55-9518-56d442d0c058","Type":"ContainerStarted","Data":"6d726b60b897f6320ee4af966929f6173375d9254b640bebae41795c0fa24687"}
Jul 28 14:26:01.909231 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:01.909196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:01.924911 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:01.924882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:02.707292 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:02.707267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:19.969555 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.969502    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:19.970870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970834    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="prometheus" containerID="cri-o://b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39" gracePeriod=600
Jul 28 14:26:19.971059 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970850    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45" gracePeriod=600
Jul 28 14:26:19.971150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970909    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="config-reloader" containerID="cri-o://7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea" gracePeriod=600
Jul 28 14:26:19.971205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970895    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-web" containerID="cri-o://70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a" gracePeriod=600
Jul 28 14:26:19.971260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970908    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="thanos-sidecar" containerID="cri-o://3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1" gracePeriod=600
Jul 28 14:26:19.971363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:19.970910    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy" containerID="cri-o://1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be" gracePeriod=600
Jul 28 14:26:20.757975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.757938    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45" exitCode=0
Jul 28 14:26:20.757975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.757964    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be" exitCode=0
Jul 28 14:26:20.757975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.757972    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1" exitCode=0
Jul 28 14:26:20.757975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.757979    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea" exitCode=0
Jul 28 14:26:20.757975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.757985    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39" exitCode=0
Jul 28 14:26:20.758341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.758010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"}
Jul 28 14:26:20.758341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.758088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"}
Jul 28 14:26:20.758341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.758146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"}
Jul 28 14:26:20.758341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.758183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"}
Jul 28 14:26:20.758341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:20.758224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"}
Jul 28 14:26:21.217002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.216980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.348205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvp5l\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348391    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.348487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.348771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db\") pod \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\" (UID: \"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9\") "
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.349042    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.349061    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.349086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.349073    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-metrics-client-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.349737 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.349414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.350661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.350353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:26:21.350661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.350583    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.351474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.351483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.352172    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config" (OuterVolumeSpecName: "config") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.352194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.352219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l" (OuterVolumeSpecName: "kube-api-access-pvp5l") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "kube-api-access-pvp5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.352419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.352588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.352478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:26:21.353333 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.353309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.353755 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.353723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.353862 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.353804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.353862 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.353808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out" (OuterVolumeSpecName: "config-out") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:26:21.353862 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.353824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.354394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.354376    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.363903 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.363883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config" (OuterVolumeSpecName: "web-config") pod "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" (UID: "f9147bc2-a56e-4ea7-8b8f-473762bf3fe9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:26:21.449580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449527    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvp5l\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-kube-api-access-pvp5l\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449573    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-tls-assets\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449584    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-grpc-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449592    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449604    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449633    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-web-config\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449642    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449651    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449660    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449668    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-kube-rbac-proxy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449677    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449685    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config-out\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449693    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-metrics-client-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449700    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-prometheus-k8s-db\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449708    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449716    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.449804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.449724    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9-config\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:26:21.764406 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.764372    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerID="70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a" exitCode=0
Jul 28 14:26:21.764602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.764434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"}
Jul 28 14:26:21.764602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.764444    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.764602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.764492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9147bc2-a56e-4ea7-8b8f-473762bf3fe9","Type":"ContainerDied","Data":"0adc6a8f01800f3b9e87fa24904aef78c544b01aa03e0240c4775cf89d726182"}
Jul 28 14:26:21.764602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.764524    2577 scope.go:117] "RemoveContainer" containerID="01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"
Jul 28 14:26:21.773109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.773086    2577 scope.go:117] "RemoveContainer" containerID="1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"
Jul 28 14:26:21.779877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.779862    2577 scope.go:117] "RemoveContainer" containerID="70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"
Jul 28 14:26:21.786693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.786673    2577 scope.go:117] "RemoveContainer" containerID="3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"
Jul 28 14:26:21.789973 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.789951    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:21.798382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.795995    2577 scope.go:117] "RemoveContainer" containerID="7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"
Jul 28 14:26:21.798382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.796844    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:21.805102 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.805081    2577 scope.go:117] "RemoveContainer" containerID="b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"
Jul 28 14:26:21.812006 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.811987    2577 scope.go:117] "RemoveContainer" containerID="15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf"
Jul 28 14:26:21.821299 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.821200    2577 scope.go:117] "RemoveContainer" containerID="01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"
Jul 28 14:26:21.821664 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.821607    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45\": container with ID starting with 01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45 not found: ID does not exist" containerID="01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"
Jul 28 14:26:21.821775 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.821674    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45"} err="failed to get container status \"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45\": rpc error: code = NotFound desc = could not find container \"01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45\": container with ID starting with 01f37a919d2b39ac17f71735e3949ba6a5f0032797a05af05f3232dbf20bfb45 not found: ID does not exist"
Jul 28 14:26:21.821775 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.821717    2577 scope.go:117] "RemoveContainer" containerID="1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"
Jul 28 14:26:21.822192 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.822164    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be\": container with ID starting with 1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be not found: ID does not exist" containerID="1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"
Jul 28 14:26:21.822297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822197    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be"} err="failed to get container status \"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be\": rpc error: code = NotFound desc = could not find container \"1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be\": container with ID starting with 1b8d55d148608e29f6e355bcddf78ca5820a3a843e7c3ac5b49ff9dca7aaf8be not found: ID does not exist"
Jul 28 14:26:21.822297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822213    2577 scope.go:117] "RemoveContainer" containerID="70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"
Jul 28 14:26:21.822573 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.822526    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a\": container with ID starting with 70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a not found: ID does not exist" containerID="70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"
Jul 28 14:26:21.822629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822578    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a"} err="failed to get container status \"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a\": rpc error: code = NotFound desc = could not find container \"70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a\": container with ID starting with 70d6e4928eeecdd7eb6e0bf88c3acd02cfd511eb34fe6d4b91bc14df4235bd8a not found: ID does not exist"
Jul 28 14:26:21.822629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822593    2577 scope.go:117] "RemoveContainer" containerID="3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"
Jul 28 14:26:21.822861 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.822846    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1\": container with ID starting with 3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1 not found: ID does not exist" containerID="3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"
Jul 28 14:26:21.822905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822865    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1"} err="failed to get container status \"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1\": rpc error: code = NotFound desc = could not find container \"3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1\": container with ID starting with 3dbb9fb148c67c7fbf695b0dd2e96a6e7d095e19f49c16c64d545e38d7aae0d1 not found: ID does not exist"
Jul 28 14:26:21.822905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.822886    2577 scope.go:117] "RemoveContainer" containerID="7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"
Jul 28 14:26:21.823146 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.823124    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea\": container with ID starting with 7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea not found: ID does not exist" containerID="7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"
Jul 28 14:26:21.823222 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823153    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea"} err="failed to get container status \"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea\": rpc error: code = NotFound desc = could not find container \"7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea\": container with ID starting with 7451d9f177e7a78c663d915fef5ac65aff4a1b7530b08070a9c96d928c1052ea not found: ID does not exist"
Jul 28 14:26:21.823222 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823174    2577 scope.go:117] "RemoveContainer" containerID="b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"
Jul 28 14:26:21.823372 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823355    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:21.823419 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.823402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39\": container with ID starting with b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39 not found: ID does not exist" containerID="b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"
Jul 28 14:26:21.823455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823419    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39"} err="failed to get container status \"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39\": rpc error: code = NotFound desc = could not find container \"b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39\": container with ID starting with b70bf8a99c83fbca9b5e0401c4f1cf068930a8611b69776f29cdf24bcd894d39 not found: ID does not exist"
Jul 28 14:26:21.823455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823436    2577 scope.go:117] "RemoveContainer" containerID="15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf"
Jul 28 14:26:21.823657 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:26:21.823642    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf\": container with ID starting with 15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf not found: ID does not exist" containerID="15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-web"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823658    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf"} err="failed to get container status \"15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf\": rpc error: code = NotFound desc = could not find container \"15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf\": container with ID starting with 15a9710c55df8f9bc71d5c1ede67541bdbe930381cfbb74f0e2c57debda992cf not found: ID does not exist"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823671    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-web"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823683    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="config-reloader"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823689    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="config-reloader"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823699    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-thanos"
Jul 28 14:26:21.823709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823705    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-thanos"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823714    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="init-config-reloader"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823720    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="init-config-reloader"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823731    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823742    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="prometheus"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823747    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="prometheus"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823753    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="thanos-sidecar"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823757    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="thanos-sidecar"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823796    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="prometheus"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823805    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-thanos"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823811    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="config-reloader"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823822    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="thanos-sidecar"
Jul 28 14:26:21.823904 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.823828    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" containerName="kube-rbac-proxy-web"
Jul 28 14:26:21.828680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.828659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.831562 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.831545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 14:26:21.831667 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.831564    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 14:26:21.831967 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.831951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 14:26:21.832053 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832018    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 14:26:21.832247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 14:26:21.832335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832231    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 14:26:21.832335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 14:26:21.832420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 14:26:21.832604 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 14:26:21.832808 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 14:26:21.832914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832896    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 14:26:21.832989 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.832969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 14:26:21.833304 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.833284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5qr0i9a34brlr\""
Jul 28 14:26:21.833389 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.833329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9tbc\""
Jul 28 14:26:21.833877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.833857    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 14:26:21.835418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.835398    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 14:26:21.839433 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.839412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 14:26:21.844617 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.844595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:21.954419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhnc\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-kube-api-access-wlhnc\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.954954 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:21.955224 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:21.954968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.055976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.055942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.055976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.055979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056173 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.055997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056173 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056173 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhnc\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-kube-api-access-wlhnc\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.056976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.056929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.057469 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.057446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.057620 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.057599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.057821 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.057797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.059198 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.059167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.059198 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.059167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.059345 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.059203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.060438 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.059902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.060438 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.059913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.060438 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.060235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.060438 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.060265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.060438 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.060388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.061112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.061091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.061248 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.061227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.061749 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.061710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.062400 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.062367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.062573 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.062554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.062652 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.062636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.069447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.069423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhnc\" (UniqueName: \"kubernetes.io/projected/c8500d7c-2bce-4cf9-958b-8d98959f0d9d-kube-api-access-wlhnc\") pod \"prometheus-k8s-0\" (UID: \"c8500d7c-2bce-4cf9-958b-8d98959f0d9d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.140718 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.140678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:26:22.282328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.282262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 14:26:22.284818 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:26:22.284790    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8500d7c_2bce_4cf9_958b_8d98959f0d9d.slice/crio-6c3ea73f9754adb0e74bd009407001af0fdf39f4d1b695a8b92e83e986415619 WatchSource:0}: Error finding container 6c3ea73f9754adb0e74bd009407001af0fdf39f4d1b695a8b92e83e986415619: Status 404 returned error can't find the container with id 6c3ea73f9754adb0e74bd009407001af0fdf39f4d1b695a8b92e83e986415619
Jul 28 14:26:22.769825 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.769781    2577 generic.go:356] "Generic (PLEG): container finished" podID="c8500d7c-2bce-4cf9-958b-8d98959f0d9d" containerID="827cd0997560d7b62aaa170956e2a82b2833401db402c91bd84260f5c8f8a2ca" exitCode=0
Jul 28 14:26:22.770031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.769885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerDied","Data":"827cd0997560d7b62aaa170956e2a82b2833401db402c91bd84260f5c8f8a2ca"}
Jul 28 14:26:22.770031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.769958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"6c3ea73f9754adb0e74bd009407001af0fdf39f4d1b695a8b92e83e986415619"}
Jul 28 14:26:22.787039 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:22.787013    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9147bc2-a56e-4ea7-8b8f-473762bf3fe9" path="/var/lib/kubelet/pods/f9147bc2-a56e-4ea7-8b8f-473762bf3fe9/volumes"
Jul 28 14:26:23.780234 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"4749394ea1ac6032113b85ba50a8741e5e34604fcc2d16eebcb2e340fbde9385"}
Jul 28 14:26:23.780730 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"a8be37e050f79ba133f1f3964f8b952490281a6d6ecb3eac59618afae65c27a8"}
Jul 28 14:26:23.780730 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"1f5a16c3ac0045a31ba74bf33afd6794dc8b503bc81749500a46d44ddacd7a53"}
Jul 28 14:26:23.780730 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"a48f670ba187245632f23c01552fc85965dbc55c63f65994cffd25c213e974c3"}
Jul 28 14:26:23.780730 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"eb3cf333fee66a16291c53a8fc188a273040cfbd8414a2d6e8bc5e2e6fca3dc6"}
Jul 28 14:26:23.780730 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.780399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8500d7c-2bce-4cf9-958b-8d98959f0d9d","Type":"ContainerStarted","Data":"a33ce4b13bf2318595428e0d4a5a31b5d8ad16290f1b6234433ad6a724e8f5fe"}
Jul 28 14:26:23.806873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:23.806809    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.786014591 podStartE2EDuration="2.806787191s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.7726ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:26:23.805716065 +0000 UTC m=+221.491794177" watchObservedRunningTime="2026-07-28 14:26:23.806787191 +0000 UTC m=+221.492865317"
Jul 28 14:26:27.141129 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:26:27.141092    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:27:22.141428 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:22.141390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:27:22.156770 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:22.156744    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:27:23.015892 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:23.015862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 14:27:42.688958 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:42.688928    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:27:42.692361 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:42.692336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:27:42.695298 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:42.695267    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:27:42.699099 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:42.699082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:27:42.699816 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:27:42.699796    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 14:29:56.598278 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.598247    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"]
Jul 28 14:29:56.601036 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.601016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.603658 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.603636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 14:29:56.603752 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.603672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:29:56.604419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.604398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 14:29:56.604489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.604434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 14:29:56.604489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.604460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jfjsq\""
Jul 28 14:29:56.604631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.604614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 14:29:56.618643 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.618621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"]
Jul 28 14:29:56.719271 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.719229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98c6\" (UniqueName: \"kubernetes.io/projected/8dbc63c8-a969-43af-a377-4fe26198da91-kube-api-access-g98c6\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.719453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.719283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.719453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.719372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8dbc63c8-a969-43af-a377-4fe26198da91-manager-config\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.719453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.719416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-metrics-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.819832 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.819797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.819832 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.819834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8dbc63c8-a969-43af-a377-4fe26198da91-manager-config\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.820085 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.819862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-metrics-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.820085 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.819908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98c6\" (UniqueName: \"kubernetes.io/projected/8dbc63c8-a969-43af-a377-4fe26198da91-kube-api-access-g98c6\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.820455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.820433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8dbc63c8-a969-43af-a377-4fe26198da91-manager-config\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.822253 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.822232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.822345 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.822290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8dbc63c8-a969-43af-a377-4fe26198da91-metrics-cert\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.836404 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.836374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98c6\" (UniqueName: \"kubernetes.io/projected/8dbc63c8-a969-43af-a377-4fe26198da91-kube-api-access-g98c6\") pod \"lws-controller-manager-7cf745d86f-br6wn\" (UID: \"8dbc63c8-a969-43af-a377-4fe26198da91\") " pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:56.911061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:56.910977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:29:57.048455 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:57.048420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"]
Jul 28 14:29:57.051261 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:29:57.051236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbc63c8_a969_43af_a377_4fe26198da91.slice/crio-461c77bd5895d6400517cd8269889e937be2dc14dc0bccc57ce065860c701412 WatchSource:0}: Error finding container 461c77bd5895d6400517cd8269889e937be2dc14dc0bccc57ce065860c701412: Status 404 returned error can't find the container with id 461c77bd5895d6400517cd8269889e937be2dc14dc0bccc57ce065860c701412
Jul 28 14:29:57.053067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:57.053053    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:29:57.570010 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:29:57.569973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn" event={"ID":"8dbc63c8-a969-43af-a377-4fe26198da91","Type":"ContainerStarted","Data":"461c77bd5895d6400517cd8269889e937be2dc14dc0bccc57ce065860c701412"}
Jul 28 14:30:00.584366 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:00.584329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn" event={"ID":"8dbc63c8-a969-43af-a377-4fe26198da91","Type":"ContainerStarted","Data":"d04f93f60fedaef2a26238e2ada1792e36bd50a169456ed39fa2dd53bde8ba8a"}
Jul 28 14:30:00.584788 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:00.584417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:30:00.603014 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:00.602959    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn" podStartSLOduration=2.074181565 podStartE2EDuration="4.602943331s" totalImagesPullingTime="2.528761766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:00.601681198 +0000 UTC m=+438.287759327" watchObservedRunningTime="2026-07-28 14:30:00.602943331 +0000 UTC m=+438.289021443"
Jul 28 14:30:03.691778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.691743    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"]
Jul 28 14:30:03.695178 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.695153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.697679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.697655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 14:30:03.697813 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.697716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 14:30:03.697813 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.697804    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 14:30:03.698031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.698015    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vfcv9\""
Jul 28 14:30:03.698082 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.698067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 14:30:03.719921 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.719888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"]
Jul 28 14:30:03.784444 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.784406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.784444 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.784441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.784731 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.784466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftgc\" (UniqueName: \"kubernetes.io/projected/4ea6be87-b5f4-491b-bb79-1322bbed538b-kube-api-access-4ftgc\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.784731 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.784585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.784731 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.784697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.885701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.885656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.885879 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.885729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.885879 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.885748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.885879 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.885776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftgc\" (UniqueName: \"kubernetes.io/projected/4ea6be87-b5f4-491b-bb79-1322bbed538b-kube-api-access-4ftgc\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.885879 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.885802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.886147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.886125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.887944 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.887914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ea6be87-b5f4-491b-bb79-1322bbed538b-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.888280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.888259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.888322 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.888259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ea6be87-b5f4-491b-bb79-1322bbed538b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:03.894214 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:03.894184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftgc\" (UniqueName: \"kubernetes.io/projected/4ea6be87-b5f4-491b-bb79-1322bbed538b-kube-api-access-4ftgc\") pod \"opendatahub-operator-controller-manager-6f457fb848-ttblg\" (UID: \"4ea6be87-b5f4-491b-bb79-1322bbed538b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:04.007601 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:04.007566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:04.144640 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:04.144557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"]
Jul 28 14:30:04.148070 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:30:04.148030    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea6be87_b5f4_491b_bb79_1322bbed538b.slice/crio-ae768f84f9a58ef1c5c7e2a4ae37793a6df7a157ce92897cacc2500dfff5a1d9 WatchSource:0}: Error finding container ae768f84f9a58ef1c5c7e2a4ae37793a6df7a157ce92897cacc2500dfff5a1d9: Status 404 returned error can't find the container with id ae768f84f9a58ef1c5c7e2a4ae37793a6df7a157ce92897cacc2500dfff5a1d9
Jul 28 14:30:04.602058 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:04.602025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg" event={"ID":"4ea6be87-b5f4-491b-bb79-1322bbed538b","Type":"ContainerStarted","Data":"ae768f84f9a58ef1c5c7e2a4ae37793a6df7a157ce92897cacc2500dfff5a1d9"}
Jul 28 14:30:07.618193 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:07.618156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg" event={"ID":"4ea6be87-b5f4-491b-bb79-1322bbed538b","Type":"ContainerStarted","Data":"be93f877b05f6fea73d32be88e47189b6668ea4c2caf7366db6e5119e7b55b9d"}
Jul 28 14:30:09.628123 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:09.628091    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ea6be87-b5f4-491b-bb79-1322bbed538b" containerID="be93f877b05f6fea73d32be88e47189b6668ea4c2caf7366db6e5119e7b55b9d" exitCode=0
Jul 28 14:30:09.628123 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:09.628135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg" event={"ID":"4ea6be87-b5f4-491b-bb79-1322bbed538b","Type":"ContainerDied","Data":"be93f877b05f6fea73d32be88e47189b6668ea4c2caf7366db6e5119e7b55b9d"}
Jul 28 14:30:10.633421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:10.633382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg" event={"ID":"4ea6be87-b5f4-491b-bb79-1322bbed538b","Type":"ContainerStarted","Data":"ff8e7efacc02a4e0c1c5c41717465e50ea921c4b4ddff438878032ad9ecaa0b7"}
Jul 28 14:30:10.633813 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:10.633618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:10.657310 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:10.657256    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg" podStartSLOduration=2.834752095 podStartE2EDuration="7.657239518s" totalImagesPullingTime="3.804230967s" totalInitContainerRuntime="1.018256456s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:10.65616979 +0000 UTC m=+448.342247903" watchObservedRunningTime="2026-07-28 14:30:10.657239518 +0000 UTC m=+448.343317630"
Jul 28 14:30:11.589979 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:11.589945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cf745d86f-br6wn"
Jul 28 14:30:21.640061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:21.639980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-ttblg"
Jul 28 14:30:24.849298 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.849263    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"]
Jul 28 14:30:24.994010 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.993973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"]
Jul 28 14:30:24.994203 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.994099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:24.997072 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.997027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 14:30:24.998260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.998240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 14:30:24.998371 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:24.998312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c9pfb\""
Jul 28 14:30:25.171863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.171769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tmp\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.171863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.171831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.171863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.171851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj55\" (UniqueName: \"kubernetes.io/projected/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-kube-api-access-4nj55\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.273218 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.273185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tmp\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.273417 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.273258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.273417 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.273285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj55\" (UniqueName: \"kubernetes.io/projected/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-kube-api-access-4nj55\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.275560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.275523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tmp\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.275692 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.275674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.284392 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.284364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj55\" (UniqueName: \"kubernetes.io/projected/d3f0aed6-a460-4914-a4f0-f87dbf300ba1-kube-api-access-4nj55\") pod \"kube-auth-proxy-746f6cf6d8-k8m7s\" (UID: \"d3f0aed6-a460-4914-a4f0-f87dbf300ba1\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.304248 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.304225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"
Jul 28 14:30:25.437213 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.437188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s"]
Jul 28 14:30:25.439752 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:30:25.439724    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f0aed6_a460_4914_a4f0_f87dbf300ba1.slice/crio-99e6e70a6e920c7aedb306ea54b613fefd8a328f22aadd44be83b4845e0fdeae WatchSource:0}: Error finding container 99e6e70a6e920c7aedb306ea54b613fefd8a328f22aadd44be83b4845e0fdeae: Status 404 returned error can't find the container with id 99e6e70a6e920c7aedb306ea54b613fefd8a328f22aadd44be83b4845e0fdeae
Jul 28 14:30:25.700161 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:25.700064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s" event={"ID":"d3f0aed6-a460-4914-a4f0-f87dbf300ba1","Type":"ContainerStarted","Data":"99e6e70a6e920c7aedb306ea54b613fefd8a328f22aadd44be83b4845e0fdeae"}
Jul 28 14:30:28.714202 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:28.714165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s" event={"ID":"d3f0aed6-a460-4914-a4f0-f87dbf300ba1","Type":"ContainerStarted","Data":"3c474c58e5bdd748b827e27989c2db1c4de96acf2b5af8e603601ca2bd58485a"}
Jul 28 14:30:28.732835 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:30:28.732787    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-k8m7s" podStartSLOduration=1.992126937 podStartE2EDuration="4.732771984s" totalImagesPullingTime="2.740645047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:28.731220058 +0000 UTC m=+466.417298172" watchObservedRunningTime="2026-07-28 14:30:28.732771984 +0000 UTC m=+466.418850096"
Jul 28 14:32:15.489990 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.489956    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"]
Jul 28 14:32:15.493230 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.493210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.495704 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.495676    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 14:32:15.495813 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.495717    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 14:32:15.495813 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.495726    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 14:32:15.496341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.496328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7hsxh\""
Jul 28 14:32:15.496395 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.496349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 14:32:15.501717 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.501304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"]
Jul 28 14:32:15.522626 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.522600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.522773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.522667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrzt\" (UniqueName: \"kubernetes.io/projected/577da352-7224-47f7-855d-dbcece2dc0f3-kube-api-access-vwrzt\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.522773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.522703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/577da352-7224-47f7-855d-dbcece2dc0f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.623287 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.623249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/577da352-7224-47f7-855d-dbcece2dc0f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.623464 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.623307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.623464 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.623365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrzt\" (UniqueName: \"kubernetes.io/projected/577da352-7224-47f7-855d-dbcece2dc0f3-kube-api-access-vwrzt\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.623563 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:32:15.623485    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 14:32:15.623608 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:32:15.623585    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert podName:577da352-7224-47f7-855d-dbcece2dc0f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:32:16.123563961 +0000 UTC m=+573.809642061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-cfkc9" (UID: "577da352-7224-47f7-855d-dbcece2dc0f3") : secret "plugin-serving-cert" not found
Jul 28 14:32:15.623930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.623911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/577da352-7224-47f7-855d-dbcece2dc0f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:15.633511 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:15.633478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrzt\" (UniqueName: \"kubernetes.io/projected/577da352-7224-47f7-855d-dbcece2dc0f3-kube-api-access-vwrzt\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:16.128327 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:16.128293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:16.130739 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:16.130714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/577da352-7224-47f7-855d-dbcece2dc0f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cfkc9\" (UID: \"577da352-7224-47f7-855d-dbcece2dc0f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:16.413938 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:16.413855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"
Jul 28 14:32:16.540623 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:16.540499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9"]
Jul 28 14:32:16.544115 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:32:16.544089    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod577da352_7224_47f7_855d_dbcece2dc0f3.slice/crio-4e31fe90820bbe76c570ae3380d558cb7c7b6965ba9198a4ec4a9af0c914b5c4 WatchSource:0}: Error finding container 4e31fe90820bbe76c570ae3380d558cb7c7b6965ba9198a4ec4a9af0c914b5c4: Status 404 returned error can't find the container with id 4e31fe90820bbe76c570ae3380d558cb7c7b6965ba9198a4ec4a9af0c914b5c4
Jul 28 14:32:17.153866 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:17.153825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9" event={"ID":"577da352-7224-47f7-855d-dbcece2dc0f3","Type":"ContainerStarted","Data":"4e31fe90820bbe76c570ae3380d558cb7c7b6965ba9198a4ec4a9af0c914b5c4"}
Jul 28 14:32:40.272034 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:40.271996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9" event={"ID":"577da352-7224-47f7-855d-dbcece2dc0f3","Type":"ContainerStarted","Data":"507b39b7089b00fc5910638afdbb5805ece0bca513251a1737395dffb6d04632"}
Jul 28 14:32:40.290146 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:40.290091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cfkc9" podStartSLOduration=2.582866718 podStartE2EDuration="25.290075365s" totalImagesPullingTime="22.707208647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:40.288190941 +0000 UTC m=+597.974269058" watchObservedRunningTime="2026-07-28 14:32:40.290075365 +0000 UTC m=+597.976153477"
Jul 28 14:32:42.718183 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:42.718148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:32:42.721311 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:42.721287    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:32:42.721552 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:42.721522    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:32:42.724774 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:32:42.724755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:33:11.682240 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.682158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:11.736430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.736394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:11.736627 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.736496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.738883 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.738862    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 14:33:11.774986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.774956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:11.788520 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.788484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhsf\" (UniqueName: \"kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.788696 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.788602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.889055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.889024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.889210 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.889085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhsf\" (UniqueName: \"kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.889666 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.889648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:11.901387 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:11.901359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhsf\" (UniqueName: \"kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf\") pod \"limitador-limitador-64c94fd64f-wg9zs\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:12.046727 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.046689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:12.179239 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.179214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:12.181632 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:12.181604    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5a4369_cd01_4d8e_9d66_f3de7004017f.slice/crio-55c805f24dc0f7da6e250b8e757397029fa22aedf3fc7b3c96ab35c521fb2478 WatchSource:0}: Error finding container 55c805f24dc0f7da6e250b8e757397029fa22aedf3fc7b3c96ab35c521fb2478: Status 404 returned error can't find the container with id 55c805f24dc0f7da6e250b8e757397029fa22aedf3fc7b3c96ab35c521fb2478
Jul 28 14:33:12.408491 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.408408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" event={"ID":"0f5a4369-cd01-4d8e-9d66-f3de7004017f","Type":"ContainerStarted","Data":"55c805f24dc0f7da6e250b8e757397029fa22aedf3fc7b3c96ab35c521fb2478"}
Jul 28 14:33:12.510741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.510705    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:12.515347 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.515322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:12.519510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.519489    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z5jp2\""
Jul 28 14:33:12.530246 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.530224    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:12.593180 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.593142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7gk\" (UniqueName: \"kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk\") pod \"authorino-58cd955675-skjxr\" (UID: \"fc6f0d4c-dad0-4a17-8a95-0285188182b2\") " pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:12.694166 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.694072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7gk\" (UniqueName: \"kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk\") pod \"authorino-58cd955675-skjxr\" (UID: \"fc6f0d4c-dad0-4a17-8a95-0285188182b2\") " pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:12.702273 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.702246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7gk\" (UniqueName: \"kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk\") pod \"authorino-58cd955675-skjxr\" (UID: \"fc6f0d4c-dad0-4a17-8a95-0285188182b2\") " pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:12.706650 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.706619    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:12.710463 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.710446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:12.715718 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.715695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:12.795221 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.795186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pvq\" (UniqueName: \"kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq\") pod \"authorino-75d556bd77-jhhgt\" (UID: \"00c2e3ee-006a-4340-a7c9-dbe794144045\") " pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:12.825155 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.825124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:12.896131 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.896097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pvq\" (UniqueName: \"kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq\") pod \"authorino-75d556bd77-jhhgt\" (UID: \"00c2e3ee-006a-4340-a7c9-dbe794144045\") " pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:12.905224 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.905171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pvq\" (UniqueName: \"kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq\") pod \"authorino-75d556bd77-jhhgt\" (UID: \"00c2e3ee-006a-4340-a7c9-dbe794144045\") " pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:12.990679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:12.990628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:12.994270 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:12.994238    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc6f0d4c_dad0_4a17_8a95_0285188182b2.slice/crio-05ad46046180ffecacada04c7be7cac85d55233ce902f9b234154ece4da7ec3e WatchSource:0}: Error finding container 05ad46046180ffecacada04c7be7cac85d55233ce902f9b234154ece4da7ec3e: Status 404 returned error can't find the container with id 05ad46046180ffecacada04c7be7cac85d55233ce902f9b234154ece4da7ec3e
Jul 28 14:33:13.020604 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:13.020568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:13.201382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:13.201289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:13.204037 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:13.204002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00c2e3ee_006a_4340_a7c9_dbe794144045.slice/crio-7f2d70aa7a0adc0c939bd2c92730882245e7e873fa57ff8aba06dc26efd23988 WatchSource:0}: Error finding container 7f2d70aa7a0adc0c939bd2c92730882245e7e873fa57ff8aba06dc26efd23988: Status 404 returned error can't find the container with id 7f2d70aa7a0adc0c939bd2c92730882245e7e873fa57ff8aba06dc26efd23988
Jul 28 14:33:13.416611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:13.416477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-skjxr" event={"ID":"fc6f0d4c-dad0-4a17-8a95-0285188182b2","Type":"ContainerStarted","Data":"05ad46046180ffecacada04c7be7cac85d55233ce902f9b234154ece4da7ec3e"}
Jul 28 14:33:13.418100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:13.418063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jhhgt" event={"ID":"00c2e3ee-006a-4340-a7c9-dbe794144045","Type":"ContainerStarted","Data":"7f2d70aa7a0adc0c939bd2c92730882245e7e873fa57ff8aba06dc26efd23988"}
Jul 28 14:33:17.442519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.442430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-skjxr" event={"ID":"fc6f0d4c-dad0-4a17-8a95-0285188182b2","Type":"ContainerStarted","Data":"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"}
Jul 28 14:33:17.444068 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.444036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jhhgt" event={"ID":"00c2e3ee-006a-4340-a7c9-dbe794144045","Type":"ContainerStarted","Data":"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"}
Jul 28 14:33:17.445419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.445391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" event={"ID":"0f5a4369-cd01-4d8e-9d66-f3de7004017f","Type":"ContainerStarted","Data":"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"}
Jul 28 14:33:17.445575 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.445556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:17.457922 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.457882    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-skjxr" podStartSLOduration=1.346709106 podStartE2EDuration="5.457868114s" totalImagesPullingTime="4.111159008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:17.456428913 +0000 UTC m=+635.142507027" watchObservedRunningTime="2026-07-28 14:33:17.457868114 +0000 UTC m=+635.143946227"
Jul 28 14:33:17.473450 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.473402    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" podStartSLOduration=1.606893898 podStartE2EDuration="6.473387407s" totalImagesPullingTime="4.866493509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:17.471898852 +0000 UTC m=+635.157976965" watchObservedRunningTime="2026-07-28 14:33:17.473387407 +0000 UTC m=+635.159465520"
Jul 28 14:33:17.486808 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.486749    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jhhgt" podStartSLOduration=1.593559324 podStartE2EDuration="5.48673087s" totalImagesPullingTime="3.893171546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:17.484858077 +0000 UTC m=+635.170936192" watchObservedRunningTime="2026-07-28 14:33:17.48673087 +0000 UTC m=+635.172808984"
Jul 28 14:33:17.515296 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:17.515262    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:19.454484 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:19.454445    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-skjxr" podUID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" containerName="authorino" containerID="cri-o://6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5" gracePeriod=30
Jul 28 14:33:19.705255 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:19.705192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:19.755103 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:19.755072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv7gk\" (UniqueName: \"kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk\") pod \"fc6f0d4c-dad0-4a17-8a95-0285188182b2\" (UID: \"fc6f0d4c-dad0-4a17-8a95-0285188182b2\") "
Jul 28 14:33:19.757163 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:19.757136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk" (OuterVolumeSpecName: "kube-api-access-fv7gk") pod "fc6f0d4c-dad0-4a17-8a95-0285188182b2" (UID: "fc6f0d4c-dad0-4a17-8a95-0285188182b2"). InnerVolumeSpecName "kube-api-access-fv7gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:33:19.856447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:19.856415    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7gk\" (UniqueName: \"kubernetes.io/projected/fc6f0d4c-dad0-4a17-8a95-0285188182b2-kube-api-access-fv7gk\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:33:20.460069 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.460041    2577 generic.go:356] "Generic (PLEG): container finished" podID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" containerID="6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5" exitCode=0
Jul 28 14:33:20.460488 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.460083    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-skjxr"
Jul 28 14:33:20.460488 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.460132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-skjxr" event={"ID":"fc6f0d4c-dad0-4a17-8a95-0285188182b2","Type":"ContainerDied","Data":"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"}
Jul 28 14:33:20.460488 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.460169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-skjxr" event={"ID":"fc6f0d4c-dad0-4a17-8a95-0285188182b2","Type":"ContainerDied","Data":"05ad46046180ffecacada04c7be7cac85d55233ce902f9b234154ece4da7ec3e"}
Jul 28 14:33:20.460488 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.460188    2577 scope.go:117] "RemoveContainer" containerID="6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"
Jul 28 14:33:20.469956 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.469940    2577 scope.go:117] "RemoveContainer" containerID="6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"
Jul 28 14:33:20.470203 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:20.470187    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5\": container with ID starting with 6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5 not found: ID does not exist" containerID="6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"
Jul 28 14:33:20.470257 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.470212    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5"} err="failed to get container status \"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5\": rpc error: code = NotFound desc = could not find container \"6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5\": container with ID starting with 6d257e5801146239be8a243ee191192e36fe55fc7b8bfaeba62fac6b311a51d5 not found: ID does not exist"
Jul 28 14:33:20.480947 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.480923    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:20.484688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.484666    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-skjxr"]
Jul 28 14:33:20.785512 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:20.785426    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" path="/var/lib/kubelet/pods/fc6f0d4c-dad0-4a17-8a95-0285188182b2/volumes"
Jul 28 14:33:26.787282 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:26.787249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:26.787784 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:26.787439    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" podUID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" containerName="limitador" containerID="cri-o://a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e" gracePeriod=30
Jul 28 14:33:26.788161 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:26.788102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:27.344092 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.344070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:27.418930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.418845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file\") pod \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") "
Jul 28 14:33:27.418930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.418885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvhsf\" (UniqueName: \"kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf\") pod \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\" (UID: \"0f5a4369-cd01-4d8e-9d66-f3de7004017f\") "
Jul 28 14:33:27.419211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.419188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file" (OuterVolumeSpecName: "config-file") pod "0f5a4369-cd01-4d8e-9d66-f3de7004017f" (UID: "0f5a4369-cd01-4d8e-9d66-f3de7004017f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:33:27.420876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.420846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf" (OuterVolumeSpecName: "kube-api-access-wvhsf") pod "0f5a4369-cd01-4d8e-9d66-f3de7004017f" (UID: "0f5a4369-cd01-4d8e-9d66-f3de7004017f"). InnerVolumeSpecName "kube-api-access-wvhsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:33:27.494551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.494510    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" containerID="a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e" exitCode=0
Jul 28 14:33:27.494705 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.494575    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"
Jul 28 14:33:27.494705 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.494600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" event={"ID":"0f5a4369-cd01-4d8e-9d66-f3de7004017f","Type":"ContainerDied","Data":"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"}
Jul 28 14:33:27.494705 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.494658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wg9zs" event={"ID":"0f5a4369-cd01-4d8e-9d66-f3de7004017f","Type":"ContainerDied","Data":"55c805f24dc0f7da6e250b8e757397029fa22aedf3fc7b3c96ab35c521fb2478"}
Jul 28 14:33:27.494803 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.494739    2577 scope.go:117] "RemoveContainer" containerID="a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"
Jul 28 14:33:27.503672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.503648    2577 scope.go:117] "RemoveContainer" containerID="a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"
Jul 28 14:33:27.503933 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:27.503907    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e\": container with ID starting with a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e not found: ID does not exist" containerID="a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"
Jul 28 14:33:27.503988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.503945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e"} err="failed to get container status \"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e\": rpc error: code = NotFound desc = could not find container \"a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e\": container with ID starting with a660de31bef88297175fce3cf189494bf05e3b6f9956b630885ccebae0811f7e not found: ID does not exist"
Jul 28 14:33:27.516234 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.516213    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:27.519526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.519504    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0f5a4369-cd01-4d8e-9d66-f3de7004017f-config-file\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:33:27.519526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.519528    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhsf\" (UniqueName: \"kubernetes.io/projected/0f5a4369-cd01-4d8e-9d66-f3de7004017f-kube-api-access-wvhsf\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:33:27.519709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:27.519691    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wg9zs"]
Jul 28 14:33:28.785104 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:28.785071    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" path="/var/lib/kubelet/pods/0f5a4369-cd01-4d8e-9d66-f3de7004017f/volumes"
Jul 28 14:33:32.732138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"]
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732406    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" containerName="limitador"
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" containerName="limitador"
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732426    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" containerName="authorino"
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732432    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" containerName="authorino"
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732497    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5a4369-cd01-4d8e-9d66-f3de7004017f" containerName="limitador"
Jul 28 14:33:32.732502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.732505    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6f0d4c-dad0-4a17-8a95-0285188182b2" containerName="authorino"
Jul 28 14:33:32.736751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.736729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.739044 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.739014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 28 14:33:32.739174 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.739133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 14:33:32.739906 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.739885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kcfxp\""
Jul 28 14:33:32.739906 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.739885    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 14:33:32.742461 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.742440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"]
Jul 28 14:33:32.863246 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.863212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2bs\" (UniqueName: \"kubernetes.io/projected/b37987e4-b840-40b6-b8e7-64e929b7a173-kube-api-access-nv2bs\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.863416 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.863258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b37987e4-b840-40b6-b8e7-64e929b7a173-data\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.964615 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.964575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2bs\" (UniqueName: \"kubernetes.io/projected/b37987e4-b840-40b6-b8e7-64e929b7a173-kube-api-access-nv2bs\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.964818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.964628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b37987e4-b840-40b6-b8e7-64e929b7a173-data\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.965077 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.965054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b37987e4-b840-40b6-b8e7-64e929b7a173-data\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:32.973061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:32.973028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2bs\" (UniqueName: \"kubernetes.io/projected/b37987e4-b840-40b6-b8e7-64e929b7a173-kube-api-access-nv2bs\") pod \"postgres-69fb4594cf-cj5nj\" (UID: \"b37987e4-b840-40b6-b8e7-64e929b7a173\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:33.048292 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:33.048216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:33.175323 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:33.175296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"]
Jul 28 14:33:33.178052 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:33.178025    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37987e4_b840_40b6_b8e7_64e929b7a173.slice/crio-36179253e3b6093c7566a08877a8eb878222a15917cb4219eff30514a66431fc WatchSource:0}: Error finding container 36179253e3b6093c7566a08877a8eb878222a15917cb4219eff30514a66431fc: Status 404 returned error can't find the container with id 36179253e3b6093c7566a08877a8eb878222a15917cb4219eff30514a66431fc
Jul 28 14:33:33.521256 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:33.521219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj" event={"ID":"b37987e4-b840-40b6-b8e7-64e929b7a173","Type":"ContainerStarted","Data":"36179253e3b6093c7566a08877a8eb878222a15917cb4219eff30514a66431fc"}
Jul 28 14:33:38.549022 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:38.548988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj" event={"ID":"b37987e4-b840-40b6-b8e7-64e929b7a173","Type":"ContainerStarted","Data":"94d2f9b4d458e7c249123f6efe6ffca4de4e52263b02611f8869adcebb6ce53c"}
Jul 28 14:33:38.549587 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:38.549051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:38.564091 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:38.564046    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj" podStartSLOduration=1.850814532 podStartE2EDuration="6.564029592s" totalImagesPullingTime="4.71321506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:38.563092711 +0000 UTC m=+656.249170824" watchObservedRunningTime="2026-07-28 14:33:38.564029592 +0000 UTC m=+656.250107704"
Jul 28 14:33:44.582000 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:44.581964    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cj5nj"
Jul 28 14:33:45.401491 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.401451    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d6474f7d-jbwz9"]
Jul 28 14:33:45.408263 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.408241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.410876 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.410852    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d6474f7d-jbwz9"]
Jul 28 14:33:45.437305 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.437268    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d6474f7d-jbwz9"]
Jul 28 14:33:45.437621 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:45.437585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mhp7s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76d6474f7d-jbwz9" podUID="1bae90b0-7705-4e39-9ca5-6a69e7bb54d8"
Jul 28 14:33:45.466011 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.465979    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:33:45.469743 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.469726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.471861 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.471844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 14:33:45.475399 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.475377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhp7s\" (UniqueName: \"kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s\") pod \"authorino-76d6474f7d-jbwz9\" (UID: \"1bae90b0-7705-4e39-9ca5-6a69e7bb54d8\") " pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.475927 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.475907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:33:45.576367 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.576330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.576367 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.576365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvps7\" (UniqueName: \"kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.576648 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.576390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhp7s\" (UniqueName: \"kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s\") pod \"authorino-76d6474f7d-jbwz9\" (UID: \"1bae90b0-7705-4e39-9ca5-6a69e7bb54d8\") " pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.579710 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.579690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.584583 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.584558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhp7s\" (UniqueName: \"kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s\") pod \"authorino-76d6474f7d-jbwz9\" (UID: \"1bae90b0-7705-4e39-9ca5-6a69e7bb54d8\") " pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.587203 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.587187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:45.677785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.677699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhp7s\" (UniqueName: \"kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s\") pod \"1bae90b0-7705-4e39-9ca5-6a69e7bb54d8\" (UID: \"1bae90b0-7705-4e39-9ca5-6a69e7bb54d8\") "
Jul 28 14:33:45.677929 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.677852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.677929 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.677877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvps7\" (UniqueName: \"kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.679740 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.679706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s" (OuterVolumeSpecName: "kube-api-access-mhp7s") pod "1bae90b0-7705-4e39-9ca5-6a69e7bb54d8" (UID: "1bae90b0-7705-4e39-9ca5-6a69e7bb54d8"). InnerVolumeSpecName "kube-api-access-mhp7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:33:45.680166 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.680148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.685459 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.685436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvps7\" (UniqueName: \"kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7\") pod \"authorino-5c7cf76794-gmhq6\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") " pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.778720 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.778678    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhp7s\" (UniqueName: \"kubernetes.io/projected/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8-kube-api-access-mhp7s\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:33:45.779472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.779454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:33:45.908306 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:45.908283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:33:45.910554 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:45.910502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f211ef7_f6bb_42a5_9a7a_508454236c77.slice/crio-2799ca747ff9ebb45dbfe54e3fc2023a01503ec7cb556d231d2573c1e2f6236d WatchSource:0}: Error finding container 2799ca747ff9ebb45dbfe54e3fc2023a01503ec7cb556d231d2573c1e2f6236d: Status 404 returned error can't find the container with id 2799ca747ff9ebb45dbfe54e3fc2023a01503ec7cb556d231d2573c1e2f6236d
Jul 28 14:33:46.586234 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.586212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6474f7d-jbwz9"
Jul 28 14:33:46.586648 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.586206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" event={"ID":"6f211ef7-f6bb-42a5-9a7a-508454236c77","Type":"ContainerStarted","Data":"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"}
Jul 28 14:33:46.586648 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.586335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" event={"ID":"6f211ef7-f6bb-42a5-9a7a-508454236c77","Type":"ContainerStarted","Data":"2799ca747ff9ebb45dbfe54e3fc2023a01503ec7cb556d231d2573c1e2f6236d"}
Jul 28 14:33:46.605820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.605766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" podStartSLOduration=1.045762568 podStartE2EDuration="1.605728102s" totalImagesPullingTime="559.965534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:46.603769293 +0000 UTC m=+664.289847407" watchObservedRunningTime="2026-07-28 14:33:46.605728102 +0000 UTC m=+664.291806214"
Jul 28 14:33:46.628013 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.627978    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d6474f7d-jbwz9"]
Jul 28 14:33:46.634851 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.634817    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:46.635561 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.635039    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jhhgt" podUID="00c2e3ee-006a-4340-a7c9-dbe794144045" containerName="authorino" containerID="cri-o://05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186" gracePeriod=30
Jul 28 14:33:46.636776 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.636751    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d6474f7d-jbwz9"]
Jul 28 14:33:46.786980 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.786946    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bae90b0-7705-4e39-9ca5-6a69e7bb54d8" path="/var/lib/kubelet/pods/1bae90b0-7705-4e39-9ca5-6a69e7bb54d8/volumes"
Jul 28 14:33:46.887168 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.887142    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:46.988265 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.988234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85pvq\" (UniqueName: \"kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq\") pod \"00c2e3ee-006a-4340-a7c9-dbe794144045\" (UID: \"00c2e3ee-006a-4340-a7c9-dbe794144045\") "
Jul 28 14:33:46.990296 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:46.990267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq" (OuterVolumeSpecName: "kube-api-access-85pvq") pod "00c2e3ee-006a-4340-a7c9-dbe794144045" (UID: "00c2e3ee-006a-4340-a7c9-dbe794144045"). InnerVolumeSpecName "kube-api-access-85pvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:33:47.088961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.088869    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pvq\" (UniqueName: \"kubernetes.io/projected/00c2e3ee-006a-4340-a7c9-dbe794144045-kube-api-access-85pvq\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:33:47.591848 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.591816    2577 generic.go:356] "Generic (PLEG): container finished" podID="00c2e3ee-006a-4340-a7c9-dbe794144045" containerID="05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186" exitCode=0
Jul 28 14:33:47.592303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.591859    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jhhgt"
Jul 28 14:33:47.592303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.591902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jhhgt" event={"ID":"00c2e3ee-006a-4340-a7c9-dbe794144045","Type":"ContainerDied","Data":"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"}
Jul 28 14:33:47.592303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.591939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jhhgt" event={"ID":"00c2e3ee-006a-4340-a7c9-dbe794144045","Type":"ContainerDied","Data":"7f2d70aa7a0adc0c939bd2c92730882245e7e873fa57ff8aba06dc26efd23988"}
Jul 28 14:33:47.592303 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.591956    2577 scope.go:117] "RemoveContainer" containerID="05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"
Jul 28 14:33:47.600358 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.600343    2577 scope.go:117] "RemoveContainer" containerID="05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"
Jul 28 14:33:47.600672 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:47.600652    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186\": container with ID starting with 05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186 not found: ID does not exist" containerID="05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"
Jul 28 14:33:47.600751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.600680    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186"} err="failed to get container status \"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186\": rpc error: code = NotFound desc = could not find container \"05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186\": container with ID starting with 05f4267a212486ad8e1c2e8e9f8840d24d74d758ecef3852dacc7f7bf4044186 not found: ID does not exist"
Jul 28 14:33:47.611034 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.610937    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:47.612930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:47.612906    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jhhgt"]
Jul 28 14:33:48.789626 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:48.789594    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c2e3ee-006a-4340-a7c9-dbe794144045" path="/var/lib/kubelet/pods/00c2e3ee-006a-4340-a7c9-dbe794144045/volumes"
Jul 28 14:33:51.620555 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.620500    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:33:51.621037 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.620979    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c2e3ee-006a-4340-a7c9-dbe794144045" containerName="authorino"
Jul 28 14:33:51.621037 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.620993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c2e3ee-006a-4340-a7c9-dbe794144045" containerName="authorino"
Jul 28 14:33:51.621148 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.621081    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c2e3ee-006a-4340-a7c9-dbe794144045" containerName="authorino"
Jul 28 14:33:51.832630 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.832592    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:33:51.832805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.832729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:51.835484 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.835460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Jul 28 14:33:51.835631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.835522    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 14:33:51.835686 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.835629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 14:33:51.836136 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.836115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h752g\""
Jul 28 14:33:51.936259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.936183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgdt\" (UniqueName: \"kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:51.936259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.936239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:51.936459 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:51.936281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.037439 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.037400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgdt\" (UniqueName: \"kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.037657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.037463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.037657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.037508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.037657 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:52.037635    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 14:33:52.037840 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:33:52.037696    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert podName:9590d003-4161-47ab-ad8d-76df3d076ebf nodeName:}" failed. No retries permitted until 2026-07-28 14:33:52.537677574 +0000 UTC m=+670.223755668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert") pod "maas-controller-77bfd6665c-nvw26" (UID: "9590d003-4161-47ab-ad8d-76df3d076ebf") : secret "maas-controller-webhook-cert" not found
Jul 28 14:33:52.040059 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.040037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.046554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.046494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgdt\" (UniqueName: \"kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.542866 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.542829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.545191 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.545165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") pod \"maas-controller-77bfd6665c-nvw26\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") " pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.743496 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.743458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:52.960341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:52.960308    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:33:52.963418 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:33:52.963395    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9590d003_4161_47ab_ad8d_76df3d076ebf.slice/crio-8c287a1c180f0c71a8eb7f2b63b4095187154ff7f350a803c3e1a8a1a730ab41 WatchSource:0}: Error finding container 8c287a1c180f0c71a8eb7f2b63b4095187154ff7f350a803c3e1a8a1a730ab41: Status 404 returned error can't find the container with id 8c287a1c180f0c71a8eb7f2b63b4095187154ff7f350a803c3e1a8a1a730ab41
Jul 28 14:33:53.622100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:53.622064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-nvw26" event={"ID":"9590d003-4161-47ab-ad8d-76df3d076ebf","Type":"ContainerStarted","Data":"8c287a1c180f0c71a8eb7f2b63b4095187154ff7f350a803c3e1a8a1a730ab41"}
Jul 28 14:33:55.634315 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:55.634279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-nvw26" event={"ID":"9590d003-4161-47ab-ad8d-76df3d076ebf","Type":"ContainerStarted","Data":"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"}
Jul 28 14:33:55.634778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:55.634362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:33:55.663127 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:33:55.663073    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77bfd6665c-nvw26" podStartSLOduration=2.137136578 podStartE2EDuration="4.663053263s" totalImagesPullingTime="2.525916685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:55.661710886 +0000 UTC m=+673.347789000" watchObservedRunningTime="2026-07-28 14:33:55.663053263 +0000 UTC m=+673.349131375"
Jul 28 14:34:06.641050 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:06.641022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:34:26.912735 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:26.912694    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8r4v2"]
Jul 28 14:34:26.916310 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:26.916286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:26.919060 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:26.919029    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 14:34:26.919187 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:26.919072    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-95snz\""
Jul 28 14:34:26.924957 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:26.924935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8r4v2"]
Jul 28 14:34:27.046919 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.046873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08dbbf8-7405-44ec-b615-725d3a87abb5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.047095 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.047003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm86p\" (UniqueName: \"kubernetes.io/projected/a08dbbf8-7405-44ec-b615-725d3a87abb5-kube-api-access-tm86p\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.148565 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.148447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm86p\" (UniqueName: \"kubernetes.io/projected/a08dbbf8-7405-44ec-b615-725d3a87abb5-kube-api-access-tm86p\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.148751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.148569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08dbbf8-7405-44ec-b615-725d3a87abb5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.149517 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.149490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08dbbf8-7405-44ec-b615-725d3a87abb5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.156848 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.156815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm86p\" (UniqueName: \"kubernetes.io/projected/a08dbbf8-7405-44ec-b615-725d3a87abb5-kube-api-access-tm86p\") pod \"payload-processing-5cc6cd444b-8r4v2\" (UID: \"a08dbbf8-7405-44ec-b615-725d3a87abb5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.227708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.227678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:27.370109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.370081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8r4v2"]
Jul 28 14:34:27.372786 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:34:27.372757    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08dbbf8_7405_44ec_b615_725d3a87abb5.slice/crio-aa2c3e172688ea1780ae8e24299bf99ef8cfe0b7bcd53e149a052ed8c299157a WatchSource:0}: Error finding container aa2c3e172688ea1780ae8e24299bf99ef8cfe0b7bcd53e149a052ed8c299157a: Status 404 returned error can't find the container with id aa2c3e172688ea1780ae8e24299bf99ef8cfe0b7bcd53e149a052ed8c299157a
Jul 28 14:34:27.783321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:27.783290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2" event={"ID":"a08dbbf8-7405-44ec-b615-725d3a87abb5","Type":"ContainerStarted","Data":"aa2c3e172688ea1780ae8e24299bf99ef8cfe0b7bcd53e149a052ed8c299157a"}
Jul 28 14:34:29.794668 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:29.794630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2" event={"ID":"a08dbbf8-7405-44ec-b615-725d3a87abb5","Type":"ContainerStarted","Data":"ab7914b650d95deb01dde0c92ef92bea1734f0558883bd28b18d21abc508f039"}
Jul 28 14:34:29.795128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:29.794753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:29.810467 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:29.810408    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2" podStartSLOduration=1.539757554 podStartE2EDuration="3.810393616s" totalImagesPullingTime="2.270636062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:34:29.80887397 +0000 UTC m=+707.494952097" watchObservedRunningTime="2026-07-28 14:34:29.810393616 +0000 UTC m=+707.496471728"
Jul 28 14:34:40.801480 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:40.801406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8r4v2"
Jul 28 14:34:41.592128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.592087    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"]
Jul 28 14:34:41.595972 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.595950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.598109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.598088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bnhwc\""
Jul 28 14:34:41.598224 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.598190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 14:34:41.598746 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.598732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 14:34:41.598819 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.598774    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 14:34:41.606359 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.606330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"]
Jul 28 14:34:41.777008 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.776970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.777008 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.777011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.777240 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.777044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwf4\" (UniqueName: \"kubernetes.io/projected/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-kube-api-access-9rwf4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.777240 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.777065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.777240 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.777081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.777240 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.777190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.878689 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwf4\" (UniqueName: \"kubernetes.io/projected/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-kube-api-access-9rwf4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.878689 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.878689 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.878828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.879220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.879252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.879280 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.879261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.881032 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.881005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.881146 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.881130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.887511 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.887490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwf4\" (UniqueName: \"kubernetes.io/projected/bfbf0720-8d09-4fdb-95e9-f9f60c71750d-kube-api-access-9rwf4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl\" (UID: \"bfbf0720-8d09-4fdb-95e9-f9f60c71750d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:41.909450 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:41.909412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:42.043000 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:42.042909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"]
Jul 28 14:34:42.046697 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:34:42.046670    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbf0720_8d09_4fdb_95e9_f9f60c71750d.slice/crio-e1ce180b3f7c96ea8850c353db6e7d3e848cfa57c40f9c1362f7d89482a11920 WatchSource:0}: Error finding container e1ce180b3f7c96ea8850c353db6e7d3e848cfa57c40f9c1362f7d89482a11920: Status 404 returned error can't find the container with id e1ce180b3f7c96ea8850c353db6e7d3e848cfa57c40f9c1362f7d89482a11920
Jul 28 14:34:42.853486 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:42.853453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl" event={"ID":"bfbf0720-8d09-4fdb-95e9-f9f60c71750d","Type":"ContainerStarted","Data":"e1ce180b3f7c96ea8850c353db6e7d3e848cfa57c40f9c1362f7d89482a11920"}
Jul 28 14:34:43.859436 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:43.859398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl" event={"ID":"bfbf0720-8d09-4fdb-95e9-f9f60c71750d","Type":"ContainerStarted","Data":"deb7dcc75a6da017309a89e7c6bf0de8b864fc9b3927786671a7e4161a5108c4"}
Jul 28 14:34:43.877618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:43.877568    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl" podStartSLOduration=1.733793496 podStartE2EDuration="2.877550973s" totalImagesPullingTime="1.143757477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:34:43.87686729 +0000 UTC m=+721.562945405" watchObservedRunningTime="2026-07-28 14:34:43.877550973 +0000 UTC m=+721.563629079"
Jul 28 14:34:51.910495 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:51.910460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:51.911134 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:51.910551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:51.922949 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:51.922922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:34:52.914008 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:34:52.913980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl"
Jul 28 14:35:17.981713 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:17.981678    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"]
Jul 28 14:35:17.987969 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:17.987948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:17.990293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:17.990270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 14:35:17.994195 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:17.994169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"]
Jul 28 14:35:18.089763 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.089943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.089943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.089943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dfc\" (UniqueName: \"kubernetes.io/projected/fecd59e1-4976-40ba-8231-0142b390b5f9-kube-api-access-w6dfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.089943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fecd59e1-4976-40ba-8231-0142b390b5f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.089943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.089897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191202 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191202 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dfc\" (UniqueName: \"kubernetes.io/projected/fecd59e1-4976-40ba-8231-0142b390b5f9-kube-api-access-w6dfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fecd59e1-4976-40ba-8231-0142b390b5f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191771 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191771 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.191943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.191837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.193642 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.193619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fecd59e1-4976-40ba-8231-0142b390b5f9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.193806 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.193791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fecd59e1-4976-40ba-8231-0142b390b5f9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.199785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.199756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dfc\" (UniqueName: \"kubernetes.io/projected/fecd59e1-4976-40ba-8231-0142b390b5f9-kube-api-access-w6dfc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zv5z7\" (UID: \"fecd59e1-4976-40ba-8231-0142b390b5f9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.299493 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.299406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:18.432750 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.432726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"]
Jul 28 14:35:18.434858 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:35:18.434827    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecd59e1_4976_40ba_8231_0142b390b5f9.slice/crio-6d992390da3630ac6b40c69bfe854e74e0b7edfb6ea25056c2c4f8315b714e73 WatchSource:0}: Error finding container 6d992390da3630ac6b40c69bfe854e74e0b7edfb6ea25056c2c4f8315b714e73: Status 404 returned error can't find the container with id 6d992390da3630ac6b40c69bfe854e74e0b7edfb6ea25056c2c4f8315b714e73
Jul 28 14:35:18.436508 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:18.436492    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:35:19.028391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:19.028356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7" event={"ID":"fecd59e1-4976-40ba-8231-0142b390b5f9","Type":"ContainerStarted","Data":"6d992390da3630ac6b40c69bfe854e74e0b7edfb6ea25056c2c4f8315b714e73"}
Jul 28 14:35:20.034607 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:20.034574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7" event={"ID":"fecd59e1-4976-40ba-8231-0142b390b5f9","Type":"ContainerStarted","Data":"763e107cee0ef04e42b16a39caedc83c1c04a205f504cb61abbde84f52da4dec"}
Jul 28 14:35:20.052580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:20.052518    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7" podStartSLOduration=1.920305004 podStartE2EDuration="3.052503858s" totalImagesPullingTime="1.132198854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:35:20.050793647 +0000 UTC m=+757.736871761" watchObservedRunningTime="2026-07-28 14:35:20.052503858 +0000 UTC m=+757.738581971"
Jul 28 14:35:28.300520 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:28.300480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:28.300975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:28.300615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:28.313587 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:28.313558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:35:29.092816 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:35:29.092788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zv5z7"
Jul 28 14:36:56.792097 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:56.792063    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:36:56.792671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:56.792240    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" podUID="6f211ef7-f6bb-42a5-9a7a-508454236c77" containerName="authorino" containerID="cri-o://40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb" gracePeriod=30
Jul 28 14:36:57.039921 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.039889    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:36:57.090648 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.090562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvps7\" (UniqueName: \"kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7\") pod \"6f211ef7-f6bb-42a5-9a7a-508454236c77\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") "
Jul 28 14:36:57.090790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.090661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert\") pod \"6f211ef7-f6bb-42a5-9a7a-508454236c77\" (UID: \"6f211ef7-f6bb-42a5-9a7a-508454236c77\") "
Jul 28 14:36:57.092626 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.092593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7" (OuterVolumeSpecName: "kube-api-access-rvps7") pod "6f211ef7-f6bb-42a5-9a7a-508454236c77" (UID: "6f211ef7-f6bb-42a5-9a7a-508454236c77"). InnerVolumeSpecName "kube-api-access-rvps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:36:57.100752 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.100723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6f211ef7-f6bb-42a5-9a7a-508454236c77" (UID: "6f211ef7-f6bb-42a5-9a7a-508454236c77"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:36:57.191822 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.191786    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvps7\" (UniqueName: \"kubernetes.io/projected/6f211ef7-f6bb-42a5-9a7a-508454236c77-kube-api-access-rvps7\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:36:57.191822 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.191816    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f211ef7-f6bb-42a5-9a7a-508454236c77-tls-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:36:57.478270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.478235    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f211ef7-f6bb-42a5-9a7a-508454236c77" containerID="40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb" exitCode=0
Jul 28 14:36:57.478454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.478279    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c7cf76794-gmhq6"
Jul 28 14:36:57.478454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.478316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" event={"ID":"6f211ef7-f6bb-42a5-9a7a-508454236c77","Type":"ContainerDied","Data":"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"}
Jul 28 14:36:57.478454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.478353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c7cf76794-gmhq6" event={"ID":"6f211ef7-f6bb-42a5-9a7a-508454236c77","Type":"ContainerDied","Data":"2799ca747ff9ebb45dbfe54e3fc2023a01503ec7cb556d231d2573c1e2f6236d"}
Jul 28 14:36:57.478454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.478371    2577 scope.go:117] "RemoveContainer" containerID="40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"
Jul 28 14:36:57.487126 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.487108    2577 scope.go:117] "RemoveContainer" containerID="40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"
Jul 28 14:36:57.487439 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:36:57.487419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb\": container with ID starting with 40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb not found: ID does not exist" containerID="40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"
Jul 28 14:36:57.487522 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.487445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb"} err="failed to get container status \"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb\": rpc error: code = NotFound desc = could not find container \"40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb\": container with ID starting with 40820ff3d8e3ea69a8399edfe93820697bd31e486341cbc42f51a1dffa5ffbbb not found: ID does not exist"
Jul 28 14:36:57.498622 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.498590    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:36:57.504868 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:57.504841    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c7cf76794-gmhq6"]
Jul 28 14:36:58.785393 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:36:58.785354    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f211ef7-f6bb-42a5-9a7a-508454236c77" path="/var/lib/kubelet/pods/6f211ef7-f6bb-42a5-9a7a-508454236c77/volumes"
Jul 28 14:37:42.747086 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:42.747007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:37:42.747902 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:42.747878    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:37:42.750823 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:42.750803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:37:42.751441 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:42.751422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:37:44.844225 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:44.844184    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:37:44.844745 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:44.844463    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77bfd6665c-nvw26" podUID="9590d003-4161-47ab-ad8d-76df3d076ebf" containerName="manager" containerID="cri-o://c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f" gracePeriod=30
Jul 28 14:37:45.096711 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.096653    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:37:45.110507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.110483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djgdt\" (UniqueName: \"kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt\") pod \"9590d003-4161-47ab-ad8d-76df3d076ebf\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") "
Jul 28 14:37:45.110606 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.110566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs\") pod \"9590d003-4161-47ab-ad8d-76df3d076ebf\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") "
Jul 28 14:37:45.110664 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.110630    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") pod \"9590d003-4161-47ab-ad8d-76df3d076ebf\" (UID: \"9590d003-4161-47ab-ad8d-76df3d076ebf\") "
Jul 28 14:37:45.112764 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.112730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt" (OuterVolumeSpecName: "kube-api-access-djgdt") pod "9590d003-4161-47ab-ad8d-76df3d076ebf" (UID: "9590d003-4161-47ab-ad8d-76df3d076ebf"). InnerVolumeSpecName "kube-api-access-djgdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:37:45.112860 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.112786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9590d003-4161-47ab-ad8d-76df3d076ebf" (UID: "9590d003-4161-47ab-ad8d-76df3d076ebf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:37:45.112860 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.112820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9590d003-4161-47ab-ad8d-76df3d076ebf" (UID: "9590d003-4161-47ab-ad8d-76df3d076ebf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:37:45.212018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.211977    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-webhook-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:37:45.212018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.212010    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgdt\" (UniqueName: \"kubernetes.io/projected/9590d003-4161-47ab-ad8d-76df3d076ebf-kube-api-access-djgdt\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:37:45.212018 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.212024    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9590d003-4161-47ab-ad8d-76df3d076ebf-metrics-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:37:45.691179 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.691143    2577 generic.go:356] "Generic (PLEG): container finished" podID="9590d003-4161-47ab-ad8d-76df3d076ebf" containerID="c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f" exitCode=0
Jul 28 14:37:45.691365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.691191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-nvw26"
Jul 28 14:37:45.691365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.691225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-nvw26" event={"ID":"9590d003-4161-47ab-ad8d-76df3d076ebf","Type":"ContainerDied","Data":"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"}
Jul 28 14:37:45.691365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.691280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-nvw26" event={"ID":"9590d003-4161-47ab-ad8d-76df3d076ebf","Type":"ContainerDied","Data":"8c287a1c180f0c71a8eb7f2b63b4095187154ff7f350a803c3e1a8a1a730ab41"}
Jul 28 14:37:45.691365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.691305    2577 scope.go:117] "RemoveContainer" containerID="c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"
Jul 28 14:37:45.700247 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.700231    2577 scope.go:117] "RemoveContainer" containerID="c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"
Jul 28 14:37:45.700488 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:37:45.700468    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f\": container with ID starting with c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f not found: ID does not exist" containerID="c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"
Jul 28 14:37:45.700576 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.700500    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f"} err="failed to get container status \"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f\": rpc error: code = NotFound desc = could not find container \"c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f\": container with ID starting with c070db76fc176fbad05b7da3d3ec9eef76cb1c42d434bb877d91a33a18dbee9f not found: ID does not exist"
Jul 28 14:37:45.711943 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.711915    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:37:45.715356 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:45.715333    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-nvw26"]
Jul 28 14:37:46.785219 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:37:46.785178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9590d003-4161-47ab-ad8d-76df3d076ebf" path="/var/lib/kubelet/pods/9590d003-4161-47ab-ad8d-76df3d076ebf/volumes"
Jul 28 14:38:46.536061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536027    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536356    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f211ef7-f6bb-42a5-9a7a-508454236c77" containerName="authorino"
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f211ef7-f6bb-42a5-9a7a-508454236c77" containerName="authorino"
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9590d003-4161-47ab-ad8d-76df3d076ebf" containerName="manager"
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9590d003-4161-47ab-ad8d-76df3d076ebf" containerName="manager"
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536460    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f211ef7-f6bb-42a5-9a7a-508454236c77" containerName="authorino"
Jul 28 14:38:46.536472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.536468    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9590d003-4161-47ab-ad8d-76df3d076ebf" containerName="manager"
Jul 28 14:38:46.540632 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.540615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.546571 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.544190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 14:38:46.546571 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.544266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 14:38:46.546571 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.544301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Jul 28 14:38:46.546571 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.544419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h752g\""
Jul 28 14:38:46.548706 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.548684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:38:46.660629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.660591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngb9\" (UniqueName: \"kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.660791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.660652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.660791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.660723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.761918 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.761888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.762090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.761933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.762090 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.761991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngb9\" (UniqueName: \"kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.764364 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.764333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.764473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.764385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.769403 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.769380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngb9\" (UniqueName: \"kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9\") pod \"maas-controller-77bfd6665c-dn8ks\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") " pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.859680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.859594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:46.988508 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:46.988486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:38:46.990894 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:38:46.990868    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0144fc63_1334_4d0f_811a_975088270886.slice/crio-7a0cf42c0a5386a0e7908eef939ec2b1c3e6895d749ad4188d460cbae3583ebb WatchSource:0}: Error finding container 7a0cf42c0a5386a0e7908eef939ec2b1c3e6895d749ad4188d460cbae3583ebb: Status 404 returned error can't find the container with id 7a0cf42c0a5386a0e7908eef939ec2b1c3e6895d749ad4188d460cbae3583ebb
Jul 28 14:38:47.970794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:47.970759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" event={"ID":"0144fc63-1334-4d0f-811a-975088270886","Type":"ContainerStarted","Data":"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"}
Jul 28 14:38:47.971182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:47.970819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" event={"ID":"0144fc63-1334-4d0f-811a-975088270886","Type":"ContainerStarted","Data":"7a0cf42c0a5386a0e7908eef939ec2b1c3e6895d749ad4188d460cbae3583ebb"}
Jul 28 14:38:47.971182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:47.970871    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:38:47.988366 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:47.988310    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" podStartSLOduration=1.451730969 podStartE2EDuration="1.988293897s" totalImagesPullingTime="536.562928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:38:47.986988086 +0000 UTC m=+965.673066213" watchObservedRunningTime="2026-07-28 14:38:47.988293897 +0000 UTC m=+965.674372009"
Jul 28 14:38:58.977783 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:38:58.977749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:39:03.638204 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:03.638153    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:39:03.638640 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:03.638346    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" podUID="0144fc63-1334-4d0f-811a-975088270886" containerName="manager" containerID="cri-o://99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0" gracePeriod=30
Jul 28 14:39:03.883357 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:03.883330    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:39:04.017418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.017389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ngb9\" (UniqueName: \"kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9\") pod \"0144fc63-1334-4d0f-811a-975088270886\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") "
Jul 28 14:39:04.017628 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.017437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs\") pod \"0144fc63-1334-4d0f-811a-975088270886\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") "
Jul 28 14:39:04.017628 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.017472    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert\") pod \"0144fc63-1334-4d0f-811a-975088270886\" (UID: \"0144fc63-1334-4d0f-811a-975088270886\") "
Jul 28 14:39:04.019653 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.019613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0144fc63-1334-4d0f-811a-975088270886" (UID: "0144fc63-1334-4d0f-811a-975088270886"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:39:04.019775 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.019682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0144fc63-1334-4d0f-811a-975088270886" (UID: "0144fc63-1334-4d0f-811a-975088270886"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:39:04.019775 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.019729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9" (OuterVolumeSpecName: "kube-api-access-6ngb9") pod "0144fc63-1334-4d0f-811a-975088270886" (UID: "0144fc63-1334-4d0f-811a-975088270886"). InnerVolumeSpecName "kube-api-access-6ngb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:39:04.045525 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.045491    2577 generic.go:356] "Generic (PLEG): container finished" podID="0144fc63-1334-4d0f-811a-975088270886" containerID="99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0" exitCode=0
Jul 28 14:39:04.045721 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.045556    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-dn8ks"
Jul 28 14:39:04.045721 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.045638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" event={"ID":"0144fc63-1334-4d0f-811a-975088270886","Type":"ContainerDied","Data":"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"}
Jul 28 14:39:04.045721 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.045693    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-dn8ks" event={"ID":"0144fc63-1334-4d0f-811a-975088270886","Type":"ContainerDied","Data":"7a0cf42c0a5386a0e7908eef939ec2b1c3e6895d749ad4188d460cbae3583ebb"}
Jul 28 14:39:04.045721 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.045719    2577 scope.go:117] "RemoveContainer" containerID="99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"
Jul 28 14:39:04.054579 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.054562    2577 scope.go:117] "RemoveContainer" containerID="99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"
Jul 28 14:39:04.054843 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:39:04.054821    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0\": container with ID starting with 99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0 not found: ID does not exist" containerID="99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"
Jul 28 14:39:04.054913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.054853    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0"} err="failed to get container status \"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0\": rpc error: code = NotFound desc = could not find container \"99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0\": container with ID starting with 99b3a402abc4b8e45fc1855c7d04adee51d2c5e8df6a922de10885173c27fbe0 not found: ID does not exist"
Jul 28 14:39:04.070817 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.070785    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:39:04.071016 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.071004    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-dn8ks"]
Jul 28 14:39:04.118919 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.118889    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngb9\" (UniqueName: \"kubernetes.io/projected/0144fc63-1334-4d0f-811a-975088270886-kube-api-access-6ngb9\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:39:04.118919 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.118914    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-metrics-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:39:04.118919 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.118926    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0144fc63-1334-4d0f-811a-975088270886-webhook-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:39:04.785122 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.785090    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0144fc63-1334-4d0f-811a-975088270886" path="/var/lib/kubelet/pods/0144fc63-1334-4d0f-811a-975088270886/volumes"
Jul 28 14:39:04.881363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.881332    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77bfd6665c-f59p8"]
Jul 28 14:39:04.881742 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.881728    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0144fc63-1334-4d0f-811a-975088270886" containerName="manager"
Jul 28 14:39:04.881789 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.881744    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0144fc63-1334-4d0f-811a-975088270886" containerName="manager"
Jul 28 14:39:04.881821 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.881799    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0144fc63-1334-4d0f-811a-975088270886" containerName="manager"
Jul 28 14:39:04.886087 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.886071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:04.889168 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.889144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 14:39:04.889297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.889196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h752g\""
Jul 28 14:39:04.889297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.889244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Jul 28 14:39:04.889297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.889268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 14:39:04.892794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:04.892773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-f59p8"]
Jul 28 14:39:05.026886 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.026847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-metrics-certs\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.027063 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.026896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fzw\" (UniqueName: \"kubernetes.io/projected/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-kube-api-access-q2fzw\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.027063 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.027012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-webhook-cert\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.127553 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.127441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fzw\" (UniqueName: \"kubernetes.io/projected/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-kube-api-access-q2fzw\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.127553 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.127501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-webhook-cert\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.127797 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.127572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-metrics-certs\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.129955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.129929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-metrics-certs\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.130073 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.130038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-webhook-cert\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.134699 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.134676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fzw\" (UniqueName: \"kubernetes.io/projected/c149fbb1-76ac-45c5-8e69-5f7ac4e942d3-kube-api-access-q2fzw\") pod \"maas-controller-77bfd6665c-f59p8\" (UID: \"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3\") " pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.197523 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.197469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:05.536032 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:05.536001    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bfd6665c-f59p8"]
Jul 28 14:39:05.538940 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:39:05.538913    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc149fbb1_76ac_45c5_8e69_5f7ac4e942d3.slice/crio-5cc3c4266d8c445bc4cc426d29eabc9889fb1695810e6f5d8cd15494fdf4da39 WatchSource:0}: Error finding container 5cc3c4266d8c445bc4cc426d29eabc9889fb1695810e6f5d8cd15494fdf4da39: Status 404 returned error can't find the container with id 5cc3c4266d8c445bc4cc426d29eabc9889fb1695810e6f5d8cd15494fdf4da39
Jul 28 14:39:06.057597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:06.057568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-f59p8" event={"ID":"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3","Type":"ContainerStarted","Data":"fb10e7a234327fc41cbeee028a5f2aabe5698e906430e0962df8729662b54e6b"}
Jul 28 14:39:06.058020 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:06.057622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bfd6665c-f59p8" event={"ID":"c149fbb1-76ac-45c5-8e69-5f7ac4e942d3","Type":"ContainerStarted","Data":"5cc3c4266d8c445bc4cc426d29eabc9889fb1695810e6f5d8cd15494fdf4da39"}
Jul 28 14:39:06.058020 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:06.057650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:39:06.076099 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:06.076049    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77bfd6665c-f59p8" podStartSLOduration=1.647761038 podStartE2EDuration="2.076032308s" totalImagesPullingTime="428.27127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:39:06.074054847 +0000 UTC m=+983.760132959" watchObservedRunningTime="2026-07-28 14:39:06.076032308 +0000 UTC m=+983.762110421"
Jul 28 14:39:17.065186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:39:17.065154    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77bfd6665c-f59p8"
Jul 28 14:42:42.774385 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:42:42.774307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:42:42.775343 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:42:42.775316    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:42:42.777631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:42:42.777609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:42:42.778641 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:42:42.778625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:47:42.801376 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:47:42.801346    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:47:42.804625 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:47:42.804604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:47:42.804625 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:47:42.804621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:47:42.810620 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:47:42.810596    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:48:06.788680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.788593    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:48:06.792075 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.792053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.794622 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.794604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c49cfed0-kserve-self-signed-certs\""
Jul 28 14:48:06.807627 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.807602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:48:06.856773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.856736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.856964 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.856794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.857015 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.856980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.857057 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.857040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.857148 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.857121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5gq\" (UniqueName: \"kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.857211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.857181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958229 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958422 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5gq\" (UniqueName: \"kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958422 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958422 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958422 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958729 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958729 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958729 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.958860 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.958827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.960783 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.960756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.961238 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.961219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:06.967076 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:06.967058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5gq\" (UniqueName: \"kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq\") pod \"e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") " pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:07.102447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:07.102363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:07.239519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:07.239494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:48:07.241647 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:48:07.241617    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06202993_d3d9_4239_a87d_842a6bdeff33.slice/crio-a1c499dfa094f8a0bf3e45e284f408d5ea73ef80b85cc62e52a9a32bccaa6ded WatchSource:0}: Error finding container a1c499dfa094f8a0bf3e45e284f408d5ea73ef80b85cc62e52a9a32bccaa6ded: Status 404 returned error can't find the container with id a1c499dfa094f8a0bf3e45e284f408d5ea73ef80b85cc62e52a9a32bccaa6ded
Jul 28 14:48:07.486198 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:07.486163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" event={"ID":"06202993-d3d9-4239-a87d-842a6bdeff33","Type":"ContainerStarted","Data":"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"}
Jul 28 14:48:07.486415 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:07.486209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" event={"ID":"06202993-d3d9-4239-a87d-842a6bdeff33","Type":"ContainerStarted","Data":"a1c499dfa094f8a0bf3e45e284f408d5ea73ef80b85cc62e52a9a32bccaa6ded"}
Jul 28 14:48:07.504137 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:07.504083    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" podStartSLOduration=1.5040673660000001 podStartE2EDuration="1.504067366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:48:07.501818395 +0000 UTC m=+1525.187896509" watchObservedRunningTime="2026-07-28 14:48:07.504067366 +0000 UTC m=+1525.190145479"
Jul 28 14:48:17.103478 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:17.103391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:17.103478 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:17.103456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:17.116377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:17.116353    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:17.541461 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:17.541428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:48:24.774962 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.774921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:48:24.778678 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.778652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.784406 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.784381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c49cfed0-kserve-self-signed-certs\""
Jul 28 14:48:24.801302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.801276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:48:24.930890 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.930858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.930890 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.930903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.931142 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.931021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.931142 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.931051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.931250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.931221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:24.931307 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:24.931258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfbx\" (UniqueName: \"kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032068 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.031986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032068 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032068 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032337 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfbx\" (UniqueName: \"kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032337 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032337 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032521 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032521 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.032683 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.032660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.034397 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.034374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.034676 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.034657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.039214 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.039190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfbx\" (UniqueName: \"kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx\") pod \"e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") " pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.090159 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.090117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:25.225293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.225269    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:48:25.227621 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:48:25.227589    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03a1535_b16f_48fe_b2a5_7209c3a50b04.slice/crio-4152ae1dff35bd3f5327e2ce61c356361fe823b704ac84ad78c03269dbb0cb2e WatchSource:0}: Error finding container 4152ae1dff35bd3f5327e2ce61c356361fe823b704ac84ad78c03269dbb0cb2e: Status 404 returned error can't find the container with id 4152ae1dff35bd3f5327e2ce61c356361fe823b704ac84ad78c03269dbb0cb2e
Jul 28 14:48:25.567702 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.567601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" event={"ID":"b03a1535-b16f-48fe-b2a5-7209c3a50b04","Type":"ContainerStarted","Data":"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"}
Jul 28 14:48:25.567702 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.567654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" event={"ID":"b03a1535-b16f-48fe-b2a5-7209c3a50b04","Type":"ContainerStarted","Data":"4152ae1dff35bd3f5327e2ce61c356361fe823b704ac84ad78c03269dbb0cb2e"}
Jul 28 14:48:25.584325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:25.584278    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" podStartSLOduration=1.584262049 podStartE2EDuration="1.584262049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:48:25.582738248 +0000 UTC m=+1543.268816362" watchObservedRunningTime="2026-07-28 14:48:25.584262049 +0000 UTC m=+1543.270340205"
Jul 28 14:48:35.090421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:35.090382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:35.090906 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:35.090433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:35.103296 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:35.103272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:35.624327 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:35.624292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:48:54.070025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:54.069990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:48:55.126736 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:48:55.126706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:49:09.089638 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.089594    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" podUID="06202993-d3d9-4239-a87d-842a6bdeff33" containerName="main" containerID="cri-o://01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc" gracePeriod=45
Jul 28 14:49:09.343019 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.342943    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:49:09.429176 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k5gq\" (UniqueName: \"kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs\") pod \"06202993-d3d9-4239-a87d-842a6bdeff33\" (UID: \"06202993-d3d9-4239-a87d-842a6bdeff33\") "
Jul 28 14:49:09.429680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429478    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home" (OuterVolumeSpecName: "home") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:09.429680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:09.429680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429673    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-home\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.429819 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429686    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-tmp-dir\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.429819 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.429732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache" (OuterVolumeSpecName: "model-cache") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:09.431502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.431475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm" (OuterVolumeSpecName: "dshm") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:09.431629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.431493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:49:09.431629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.431547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq" (OuterVolumeSpecName: "kube-api-access-6k5gq") pod "06202993-d3d9-4239-a87d-842a6bdeff33" (UID: "06202993-d3d9-4239-a87d-842a6bdeff33"). InnerVolumeSpecName "kube-api-access-6k5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:49:09.530165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.530125    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-dshm\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.530165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.530158    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5gq\" (UniqueName: \"kubernetes.io/projected/06202993-d3d9-4239-a87d-842a6bdeff33-kube-api-access-6k5gq\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.530165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.530172    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06202993-d3d9-4239-a87d-842a6bdeff33-model-cache\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.530400 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.530185    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06202993-d3d9-4239-a87d-842a6bdeff33-tls-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:09.769184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.769146    2577 generic.go:356] "Generic (PLEG): container finished" podID="06202993-d3d9-4239-a87d-842a6bdeff33" containerID="01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc" exitCode=0
Jul 28 14:49:09.769351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.769191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"
Jul 28 14:49:09.769351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.769205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" event={"ID":"06202993-d3d9-4239-a87d-842a6bdeff33","Type":"ContainerDied","Data":"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"}
Jul 28 14:49:09.769351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.769243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6" event={"ID":"06202993-d3d9-4239-a87d-842a6bdeff33","Type":"ContainerDied","Data":"a1c499dfa094f8a0bf3e45e284f408d5ea73ef80b85cc62e52a9a32bccaa6ded"}
Jul 28 14:49:09.769351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.769265    2577 scope.go:117] "RemoveContainer" containerID="01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"
Jul 28 14:49:09.778289 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.778274    2577 scope.go:117] "RemoveContainer" containerID="01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"
Jul 28 14:49:09.778604 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:49:09.778581    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc\": container with ID starting with 01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc not found: ID does not exist" containerID="01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"
Jul 28 14:49:09.778665 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.778613    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc"} err="failed to get container status \"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc\": rpc error: code = NotFound desc = could not find container \"01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc\": container with ID starting with 01b7815235c10a79a02fbb8e22e8a2b94a295daa9483913acda7000027c1e5bc not found: ID does not exist"
Jul 28 14:49:09.791566 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.790837    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:49:09.796123 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:09.796097    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c49cfed0-kserve-5dc45d5487-v7kk6"]
Jul 28 14:49:10.146228 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.146137    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" podUID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" containerName="main" containerID="cri-o://aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b" gracePeriod=45
Jul 28 14:49:10.401160 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.401095    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:49:10.435092 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.435270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.435270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktfbx\" (UniqueName: \"kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.435270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.435270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.435491 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435368    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:10.435565 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:10.435639 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435618    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-model-cache\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.435690 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.435649    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tmp-dir\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.437187 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.437164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm" (OuterVolumeSpecName: "dshm") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:10.437334 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.437285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:49:10.437632 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.437616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx" (OuterVolumeSpecName: "kube-api-access-ktfbx") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "kube-api-access-ktfbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:49:10.536748 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.536714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home\") pod \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\" (UID: \"b03a1535-b16f-48fe-b2a5-7209c3a50b04\") "
Jul 28 14:49:10.536935 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.536921    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a1535-b16f-48fe-b2a5-7209c3a50b04-tls-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.537006 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.536938    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfbx\" (UniqueName: \"kubernetes.io/projected/b03a1535-b16f-48fe-b2a5-7209c3a50b04-kube-api-access-ktfbx\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.537006 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.536947    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-dshm\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.537097 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.537005    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home" (OuterVolumeSpecName: "home") pod "b03a1535-b16f-48fe-b2a5-7209c3a50b04" (UID: "b03a1535-b16f-48fe-b2a5-7209c3a50b04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:49:10.637583 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.637521    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a1535-b16f-48fe-b2a5-7209c3a50b04-home\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:49:10.775619 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.775583    2577 generic.go:356] "Generic (PLEG): container finished" podID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" containerID="aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b" exitCode=0
Jul 28 14:49:10.775812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.775631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"
Jul 28 14:49:10.775812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.775679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" event={"ID":"b03a1535-b16f-48fe-b2a5-7209c3a50b04","Type":"ContainerDied","Data":"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"}
Jul 28 14:49:10.775812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.775719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc" event={"ID":"b03a1535-b16f-48fe-b2a5-7209c3a50b04","Type":"ContainerDied","Data":"4152ae1dff35bd3f5327e2ce61c356361fe823b704ac84ad78c03269dbb0cb2e"}
Jul 28 14:49:10.775812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.775737    2577 scope.go:117] "RemoveContainer" containerID="aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"
Jul 28 14:49:10.784891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.784779    2577 scope.go:117] "RemoveContainer" containerID="aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"
Jul 28 14:49:10.785168 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:49:10.785128    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b\": container with ID starting with aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b not found: ID does not exist" containerID="aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"
Jul 28 14:49:10.785288 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.785167    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b"} err="failed to get container status \"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b\": rpc error: code = NotFound desc = could not find container \"aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b\": container with ID starting with aa80dcbcb6ef0c74dbc48b1e2fb02f24eefe2bcbb64b6b4f88d1699cb42da55b not found: ID does not exist"
Jul 28 14:49:10.786518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.786499    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06202993-d3d9-4239-a87d-842a6bdeff33" path="/var/lib/kubelet/pods/06202993-d3d9-4239-a87d-842a6bdeff33/volumes"
Jul 28 14:49:10.797180 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.797157    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:49:10.800690 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:10.800668    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c49cfed0-kserve-5fb69668b7-9ztfc"]
Jul 28 14:49:12.786558 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:49:12.786511    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" path="/var/lib/kubelet/pods/b03a1535-b16f-48fe-b2a5-7209c3a50b04/volumes"
Jul 28 14:52:42.831014 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:42.830984    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:52:42.834578 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:42.834555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:52:42.836079 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:42.836061    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:52:42.841067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:42.840397    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:52:48.237949 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.237902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:52:48.238431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238421    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" containerName="main"
Jul 28 14:52:48.238489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238436    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" containerName="main"
Jul 28 14:52:48.238489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238453    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06202993-d3d9-4239-a87d-842a6bdeff33" containerName="main"
Jul 28 14:52:48.238489 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06202993-d3d9-4239-a87d-842a6bdeff33" containerName="main"
Jul 28 14:52:48.238629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238562    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06202993-d3d9-4239-a87d-842a6bdeff33" containerName="main"
Jul 28 14:52:48.238629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.238573    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03a1535-b16f-48fe-b2a5-7209c3a50b04" containerName="main"
Jul 28 14:52:48.241728 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.241708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.244084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.244063    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-856ad4e0-openshift-default-dockercfg-c8vrw\""
Jul 28 14:52:48.244084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.244074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 14:52:48.260429 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.260407    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:52:48.385343 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fqs\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.385778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.385711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fqs\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.486914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.486956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.487090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.487223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.487242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487522 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.487501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.487621 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.487499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.489320 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.489267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.489659 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.489640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.495681 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.495652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.495789 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.495769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fqs\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs\") pod \"e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") " pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.552785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.552747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:48.691979 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.691952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:52:48.693762 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:52:48.693738    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d0b470_e3bb_4dec_8b5e_d4fa6f834746.slice/crio-9f152d380fedc359db38559f24f27a8da3046ec6181312a6a2fbc549ea4a53aa WatchSource:0}: Error finding container 9f152d380fedc359db38559f24f27a8da3046ec6181312a6a2fbc549ea4a53aa: Status 404 returned error can't find the container with id 9f152d380fedc359db38559f24f27a8da3046ec6181312a6a2fbc549ea4a53aa
Jul 28 14:52:48.695816 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.695795    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:52:48.751812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:48.751724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerStarted","Data":"9f152d380fedc359db38559f24f27a8da3046ec6181312a6a2fbc549ea4a53aa"}
Jul 28 14:52:51.252558 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:51.252494    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:52:51.252839 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:51.252595    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:52:51.252839 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:51.252625    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:52:51.774388 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:51.774353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerStarted","Data":"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"}
Jul 28 14:52:51.796684 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:51.796625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podStartSLOduration=1.240423745 podStartE2EDuration="3.796610048s" totalImagesPullingTime="2.556186303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:52:51.794335277 +0000 UTC m=+1809.480413391" watchObservedRunningTime="2026-07-28 14:52:51.796610048 +0000 UTC m=+1809.482688161"
Jul 28 14:52:52.553398 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:52.553356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:52.558139 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:52.558110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:52.779130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:52.779102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:52.785579 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:52.785549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:52:53.210094 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.210062    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:52:53.213940 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.213914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.216597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.216569    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-856ad4e0-serving-cert\""
Jul 28 14:52:53.216718 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.216583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:52:53.223855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.222880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:52:53.337525 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.337488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.337753 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.337612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gl9\" (UniqueName: \"kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.337753 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.337701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.439051 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.439010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.439255 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.439075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gl9\" (UniqueName: \"kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.439255 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.439123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.439255 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:52:53.439248    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-serving-cert: secret "maas-api-e2e-ait-856ad4e0-serving-cert" not found
Jul 28 14:52:53.439422 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:52:53.439328    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls podName:6071e1d2-f1ce-44e4-8092-ba2cc9f480a3 nodeName:}" failed. No retries permitted until 2026-07-28 14:52:53.939307218 +0000 UTC m=+1811.625385324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls") pod "maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" (UID: "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3") : secret "maas-api-e2e-ait-856ad4e0-serving-cert" not found
Jul 28 14:52:53.439673 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.439653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.447430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.447406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gl9\" (UniqueName: \"kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.933966 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.933932    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:52:53.937574 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.937558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:53.939883 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.939863    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-856ad4e0\""
Jul 28 14:52:53.940015 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.939863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-856ad4e0-dockercfg-l59ts\""
Jul 28 14:52:53.944091 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.944056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.946741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.946721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:52:53.947017 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.946997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") pod \"maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:53.977060 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.977026    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:52:53.980364 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.980345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:53.991004 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:53.990978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:52:54.045121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.045088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.045277 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.045127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xq77\" (UniqueName: \"kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.126166 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.126132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:54.146633 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.146603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5xb\" (UniqueName: \"kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.146752 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.146653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.146752 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.146727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xq77\" (UniqueName: \"kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.146988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.146783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.147369 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.147345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.154840 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.154814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xq77\" (UniqueName: \"kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77\") pod \"payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.248158 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.248120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5xb\" (UniqueName: \"kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.248680 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.248194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.248941 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.248919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.250045 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.250022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.255778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.255752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5xb\" (UniqueName: \"kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb\") pod \"payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") " pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.264458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.264433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:52:54.266202 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:52:54.266175    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6071e1d2_f1ce_44e4_8092_ba2cc9f480a3.slice/crio-e13a7022c64dc3ec7e8c67a84055b57da422978165eb2d2230d26d0c4bf611bf WatchSource:0}: Error finding container e13a7022c64dc3ec7e8c67a84055b57da422978165eb2d2230d26d0c4bf611bf: Status 404 returned error can't find the container with id e13a7022c64dc3ec7e8c67a84055b57da422978165eb2d2230d26d0c4bf611bf
Jul 28 14:52:54.291810 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.291770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.395292 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.395262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:52:54.397574 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:52:54.397526    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fc38e0_9a77_4818_a911_acf222949850.slice/crio-57034bf0764b58854c7b9921622c434da072daa574241ceda8545eeea59e169f WatchSource:0}: Error finding container 57034bf0764b58854c7b9921622c434da072daa574241ceda8545eeea59e169f: Status 404 returned error can't find the container with id 57034bf0764b58854c7b9921622c434da072daa574241ceda8545eeea59e169f
Jul 28 14:52:54.452868 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.452772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:52:54.455151 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:52:54.455111    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372f35ed_eca7_4b36_bbc1_d333d8f82fff.slice/crio-acf9517e6395c010900a466ac7a1966332a635955648e880444b4299eb3b7135 WatchSource:0}: Error finding container acf9517e6395c010900a466ac7a1966332a635955648e880444b4299eb3b7135: Status 404 returned error can't find the container with id acf9517e6395c010900a466ac7a1966332a635955648e880444b4299eb3b7135
Jul 28 14:52:54.788210 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.788103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" event={"ID":"372f35ed-eca7-4b36-bbc1-d333d8f82fff","Type":"ContainerStarted","Data":"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"}
Jul 28 14:52:54.788210 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.788160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" event={"ID":"372f35ed-eca7-4b36-bbc1-d333d8f82fff","Type":"ContainerStarted","Data":"acf9517e6395c010900a466ac7a1966332a635955648e880444b4299eb3b7135"}
Jul 28 14:52:54.788447 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.788231    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:52:54.789300 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.789273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" event={"ID":"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3","Type":"ContainerStarted","Data":"e13a7022c64dc3ec7e8c67a84055b57da422978165eb2d2230d26d0c4bf611bf"}
Jul 28 14:52:54.790627 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.790600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" event={"ID":"02fc38e0-9a77-4818-a911-acf222949850","Type":"ContainerStarted","Data":"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"}
Jul 28 14:52:54.790770 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.790640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" event={"ID":"02fc38e0-9a77-4818-a911-acf222949850","Type":"ContainerStarted","Data":"57034bf0764b58854c7b9921622c434da072daa574241ceda8545eeea59e169f"}
Jul 28 14:52:54.790834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.790820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:52:54.805298 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.805250    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" podStartSLOduration=1.805237409 podStartE2EDuration="1.805237409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:52:54.803208622 +0000 UTC m=+1812.489286736" watchObservedRunningTime="2026-07-28 14:52:54.805237409 +0000 UTC m=+1812.491315499"
Jul 28 14:52:54.819400 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:54.819342    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" podStartSLOduration=1.819322093 podStartE2EDuration="1.819322093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:52:54.817561914 +0000 UTC m=+1812.503640028" watchObservedRunningTime="2026-07-28 14:52:54.819322093 +0000 UTC m=+1812.505400215"
Jul 28 14:52:55.798420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:55.798386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" event={"ID":"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3","Type":"ContainerStarted","Data":"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"}
Jul 28 14:52:55.798852 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:55.798472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:52:55.815700 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:52:55.815641    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" podStartSLOduration=1.380814588 podStartE2EDuration="2.815624719s" totalImagesPullingTime="1.434810131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:52:55.813223274 +0000 UTC m=+1813.499301401" watchObservedRunningTime="2026-07-28 14:52:55.815624719 +0000 UTC m=+1813.501702843"
Jul 28 14:53:01.808109 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:01.808079    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:53:04.850418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.850310    2577 generic.go:356] "Generic (PLEG): container finished" podID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerID="a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1" exitCode=0
Jul 28 14:53:04.850418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.850372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerDied","Data":"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"}
Jul 28 14:53:04.851031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.850890    2577 scope.go:117] "RemoveContainer" containerID="a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"
Jul 28 14:53:04.852836 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.852809    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:04.852909 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.852890    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:04.852946 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:04.852939    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:05.801100 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:05.801068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:53:05.801274 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:05.801140    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:53:05.866666 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:05.866630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerStarted","Data":"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"}
Jul 28 14:53:06.553154 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:06.553116    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:06.554365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:06.554335    2577 patch_prober.go:28] interesting pod/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 28 14:53:06.554508 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:06.554395    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 28 14:53:07.553891 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:07.553842    2577 patch_prober.go:28] interesting pod/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 28 14:53:07.554299 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:07.553913    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 28 14:53:07.780019 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:07.779958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:08.554117 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:08.554083    2577 patch_prober.go:28] interesting pod/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 28 14:53:08.554501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:08.554142    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 28 14:53:09.553947 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:09.553911    2577 patch_prober.go:28] interesting pod/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 28 14:53:09.554142 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:09.553965    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 28 14:53:10.558133 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:10.558103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:10.558982 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:10.558964    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:33.914517 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.914468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:53:33.915110 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.914746    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" podUID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" containerName="maas-api" containerID="cri-o://6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0" gracePeriod=30
Jul 28 14:53:33.967490 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.967454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:53:33.967741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.967715    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" podUID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" containerName="payload-processing" containerID="cri-o://e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5" gracePeriod=30
Jul 28 14:53:33.973930 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.973894    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:53:33.974217 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:33.974160    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" podUID="02fc38e0-9a77-4818-a911-acf222949850" containerName="payload-pre-processing" containerID="cri-o://fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285" gracePeriod=30
Jul 28 14:53:34.253431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.253396    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:53:34.253727 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.253699    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy" containerID="cri-o://24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d" gracePeriod=30
Jul 28 14:53:34.277340 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.277318    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:53:34.281138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.281119    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:53:34.285035 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.285011    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:53:34.427257 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume\") pod \"02fc38e0-9a77-4818-a911-acf222949850\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca\") pod \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gl9\" (UniqueName: \"kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9\") pod \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") pod \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\" (UID: \"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xq77\" (UniqueName: \"kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77\") pod \"02fc38e0-9a77-4818-a911-acf222949850\" (UID: \"02fc38e0-9a77-4818-a911-acf222949850\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume\") pod \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") "
Jul 28 14:53:34.427471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427469    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w5xb\" (UniqueName: \"kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb\") pod \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\" (UID: \"372f35ed-eca7-4b36-bbc1-d333d8f82fff\") "
Jul 28 14:53:34.427818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02fc38e0-9a77-4818-a911-acf222949850" (UID: "02fc38e0-9a77-4818-a911-acf222949850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:53:34.427818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" (UID: "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:53:34.427936 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.427856    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.428069 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.428048    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fc38e0-9a77-4818-a911-acf222949850-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.428189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.428162    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "372f35ed-eca7-4b36-bbc1-d333d8f82fff" (UID: "372f35ed-eca7-4b36-bbc1-d333d8f82fff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:53:34.429823 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.429793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb" (OuterVolumeSpecName: "kube-api-access-5w5xb") pod "372f35ed-eca7-4b36-bbc1-d333d8f82fff" (UID: "372f35ed-eca7-4b36-bbc1-d333d8f82fff"). InnerVolumeSpecName "kube-api-access-5w5xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:53:34.429913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.429836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77" (OuterVolumeSpecName: "kube-api-access-4xq77") pod "02fc38e0-9a77-4818-a911-acf222949850" (UID: "02fc38e0-9a77-4818-a911-acf222949850"). InnerVolumeSpecName "kube-api-access-4xq77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:53:34.429913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.429882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" (UID: "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:53:34.429913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.429890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9" (OuterVolumeSpecName: "kube-api-access-q7gl9") pod "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" (UID: "6071e1d2-f1ce-44e4-8092-ba2cc9f480a3"). InnerVolumeSpecName "kube-api-access-q7gl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:53:34.529331 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.529290    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gl9\" (UniqueName: \"kubernetes.io/projected/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-kube-api-access-q7gl9\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.529331 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.529325    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.529331 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.529338    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xq77\" (UniqueName: \"kubernetes.io/projected/02fc38e0-9a77-4818-a911-acf222949850-kube-api-access-4xq77\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.529618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.529350    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f35ed-eca7-4b36-bbc1-d333d8f82fff-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:34.529618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:34.529362    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5xb\" (UniqueName: \"kubernetes.io/projected/372f35ed-eca7-4b36-bbc1-d333d8f82fff-kube-api-access-5w5xb\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:35.002021 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.001987    2577 generic.go:356] "Generic (PLEG): container finished" podID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" containerID="e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5" exitCode=0
Jul 28 14:53:35.002496 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.002058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"
Jul 28 14:53:35.002496 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.002065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" event={"ID":"372f35ed-eca7-4b36-bbc1-d333d8f82fff","Type":"ContainerDied","Data":"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"}
Jul 28 14:53:35.002496 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.002168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b" event={"ID":"372f35ed-eca7-4b36-bbc1-d333d8f82fff","Type":"ContainerDied","Data":"acf9517e6395c010900a466ac7a1966332a635955648e880444b4299eb3b7135"}
Jul 28 14:53:35.002496 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.002191    2577 scope.go:117] "RemoveContainer" containerID="e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"
Jul 28 14:53:35.003685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.003478    2577 generic.go:356] "Generic (PLEG): container finished" podID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" containerID="6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0" exitCode=0
Jul 28 14:53:35.003685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.003530    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"
Jul 28 14:53:35.003685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.003587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" event={"ID":"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3","Type":"ContainerDied","Data":"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"}
Jul 28 14:53:35.003685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.003644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c" event={"ID":"6071e1d2-f1ce-44e4-8092-ba2cc9f480a3","Type":"ContainerDied","Data":"e13a7022c64dc3ec7e8c67a84055b57da422978165eb2d2230d26d0c4bf611bf"}
Jul 28 14:53:35.005288 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.005267    2577 generic.go:356] "Generic (PLEG): container finished" podID="02fc38e0-9a77-4818-a911-acf222949850" containerID="fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285" exitCode=0
Jul 28 14:53:35.005404 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.005311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" event={"ID":"02fc38e0-9a77-4818-a911-acf222949850","Type":"ContainerDied","Data":"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"}
Jul 28 14:53:35.005404 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.005353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz" event={"ID":"02fc38e0-9a77-4818-a911-acf222949850","Type":"ContainerDied","Data":"57034bf0764b58854c7b9921622c434da072daa574241ceda8545eeea59e169f"}
Jul 28 14:53:35.005518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.005440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"
Jul 28 14:53:35.013032 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.013014    2577 scope.go:117] "RemoveContainer" containerID="e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"
Jul 28 14:53:35.013375 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:53:35.013355    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5\": container with ID starting with e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5 not found: ID does not exist" containerID="e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"
Jul 28 14:53:35.013435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.013385    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5"} err="failed to get container status \"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5\": rpc error: code = NotFound desc = could not find container \"e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5\": container with ID starting with e355b8f5e4ec3f855a2bf15e3b217e5eca36e2cfea9d52a39e3c8217236440b5 not found: ID does not exist"
Jul 28 14:53:35.013435 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.013402    2577 scope.go:117] "RemoveContainer" containerID="6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"
Jul 28 14:53:35.022134 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.022112    2577 scope.go:117] "RemoveContainer" containerID="6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"
Jul 28 14:53:35.022430 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:53:35.022408    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0\": container with ID starting with 6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0 not found: ID does not exist" containerID="6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"
Jul 28 14:53:35.022480 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.022440    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0"} err="failed to get container status \"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0\": rpc error: code = NotFound desc = could not find container \"6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0\": container with ID starting with 6f00f34f6b09e3d4b7ffacb365bb145a7eecb8fca5fdc94997261deb8b5b0cc0 not found: ID does not exist"
Jul 28 14:53:35.022480 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.022458    2577 scope.go:117] "RemoveContainer" containerID="fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"
Jul 28 14:53:35.028521 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.028495    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:53:35.031295 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.031273    2577 scope.go:117] "RemoveContainer" containerID="fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"
Jul 28 14:53:35.031634 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:53:35.031613    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285\": container with ID starting with fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285 not found: ID does not exist" containerID="fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"
Jul 28 14:53:35.031700 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.031643    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285"} err="failed to get container status \"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285\": rpc error: code = NotFound desc = could not find container \"fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285\": container with ID starting with fecae2facd6dc3f38da643aa68455029cb1c618200ea37cfe0e2786226696285 not found: ID does not exist"
Jul 28 14:53:35.035974 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.035946    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-856ad4e0-cf4b74b-glx2b"]
Jul 28 14:53:35.045818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.045730    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:53:35.049770 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.049740    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-856ad4e0-8549dd8c44-w456c"]
Jul 28 14:53:35.059815 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.059782    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:53:35.063650 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.063620    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-856ad4e0-64c854c764-ddzcz"]
Jul 28 14:53:35.414742 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.414635    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:53:35.415238 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02fc38e0-9a77-4818-a911-acf222949850" containerName="payload-pre-processing"
Jul 28 14:53:35.415342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415242    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02fc38e0-9a77-4818-a911-acf222949850" containerName="payload-pre-processing"
Jul 28 14:53:35.415342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415274    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" containerName="payload-processing"
Jul 28 14:53:35.415342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415283    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" containerName="payload-processing"
Jul 28 14:53:35.415342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415302    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" containerName="maas-api"
Jul 28 14:53:35.415342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" containerName="maas-api"
Jul 28 14:53:35.415643 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" containerName="payload-processing"
Jul 28 14:53:35.415643 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415417    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" containerName="maas-api"
Jul 28 14:53:35.415643 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.415428    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02fc38e0-9a77-4818-a911-acf222949850" containerName="payload-pre-processing"
Jul 28 14:53:35.420007 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.419987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.422693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.422675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4033a9f3-gw-openshift-default-dockercfg-p58fk\""
Jul 28 14:53:35.428039 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.428005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:53:35.538505 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538505 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538759 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538920 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q26w\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.538920 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.538820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.639820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.639820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.639990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640074 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q26w\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640431 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640583 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.640729 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.640709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.642333 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.642313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.642421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.642403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.647626 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.647606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.647811 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.647795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q26w\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w\") pod \"e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") " pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.731411 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.731380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:35.865320 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.865294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:53:35.867353 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:53:35.867327    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569cf044_9537_4a43_824a_c47042d227ca.slice/crio-7074de4da91a4fc0181ceba8beaa973fb1c0e2f50458ddd6d632c1e00010f022 WatchSource:0}: Error finding container 7074de4da91a4fc0181ceba8beaa973fb1c0e2f50458ddd6d632c1e00010f022: Status 404 returned error can't find the container with id 7074de4da91a4fc0181ceba8beaa973fb1c0e2f50458ddd6d632c1e00010f022
Jul 28 14:53:35.869429 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.869397    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:35.869528 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.869478    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:35.869630 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:35.869523    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:36.012359 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.012250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerStarted","Data":"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"}
Jul 28 14:53:36.012359 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.012312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerStarted","Data":"7074de4da91a4fc0181ceba8beaa973fb1c0e2f50458ddd6d632c1e00010f022"}
Jul 28 14:53:36.031783 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.031694    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podStartSLOduration=1.031672092 podStartE2EDuration="1.031672092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:53:36.030287501 +0000 UTC m=+1853.716365615" watchObservedRunningTime="2026-07-28 14:53:36.031672092 +0000 UTC m=+1853.717750205"
Jul 28 14:53:36.732516 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.732483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:36.737725 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.737700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:36.786469 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.786437    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02fc38e0-9a77-4818-a911-acf222949850" path="/var/lib/kubelet/pods/02fc38e0-9a77-4818-a911-acf222949850/volumes"
Jul 28 14:53:36.786837 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.786824    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372f35ed-eca7-4b36-bbc1-d333d8f82fff" path="/var/lib/kubelet/pods/372f35ed-eca7-4b36-bbc1-d333d8f82fff/volumes"
Jul 28 14:53:36.787162 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:36.787150    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6071e1d2-f1ce-44e4-8092-ba2cc9f480a3" path="/var/lib/kubelet/pods/6071e1d2-f1ce-44e4-8092-ba2cc9f480a3/volumes"
Jul 28 14:53:37.022636 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.022003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:37.026117 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.026085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:37.582351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.582308    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:53:37.587362 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.587334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.589458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.589432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4033a9f3-serving-cert\""
Jul 28 14:53:37.589620 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.589520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:53:37.597408 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.597377    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:53:37.760831 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.760792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.761055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.760837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.761055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.760990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbh76\" (UniqueName: \"kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.861953 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.861847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbh76\" (UniqueName: \"kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.861953 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.861923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.861953 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.861944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.862611 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.862574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.864385 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.864364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.869632 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.869610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbh76\" (UniqueName: \"kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76\") pod \"maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:37.901551 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:37.901492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:38.067328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.067287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:53:38.069462 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:53:38.069438    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbe4caf_df98_42d4_93ce_ca41eb78c966.slice/crio-59d85994ad7809a030d44005afb6f5f161e0875213fd8e00e8e66104b4df4f51 WatchSource:0}: Error finding container 59d85994ad7809a030d44005afb6f5f161e0875213fd8e00e8e66104b4df4f51: Status 404 returned error can't find the container with id 59d85994ad7809a030d44005afb6f5f161e0875213fd8e00e8e66104b4df4f51
Jul 28 14:53:38.211187 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.211150    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:53:38.215768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.215745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.217845 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.217819    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4033a9f3\""
Jul 28 14:53:38.227127 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.227095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:53:38.249088 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.249056    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:53:38.253025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.253004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.255102 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.255078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4033a9f3-dockercfg-c2vjz\""
Jul 28 14:53:38.263980 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.263954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:53:38.368671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.368601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.368671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.368671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jh96\" (UniqueName: \"kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.368927 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.368752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.368927 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.368779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bbc\" (UniqueName: \"kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.469905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.469800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.469905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.469852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh96\" (UniqueName: \"kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.469905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.469903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.470182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.469926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bbc\" (UniqueName: \"kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.470777 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.470751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.470777 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.470772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.477189 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.477158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bbc\" (UniqueName: \"kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc\") pod \"payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.477321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.477170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh96\" (UniqueName: \"kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96\") pod \"payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") " pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.528472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.528441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:38.564288 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.564257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:38.883369 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.883339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:53:38.885422 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:53:38.885390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361be0c7_00df_45db_81ef_5cd85e2ff85e.slice/crio-295c07178c56d3d55242e704ac3a00171802ae3f0aa8ceadb1585d801ce34587 WatchSource:0}: Error finding container 295c07178c56d3d55242e704ac3a00171802ae3f0aa8ceadb1585d801ce34587: Status 404 returned error can't find the container with id 295c07178c56d3d55242e704ac3a00171802ae3f0aa8ceadb1585d801ce34587
Jul 28 14:53:38.919913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:38.919884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:53:38.922837 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:53:38.922808    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2219b07b_41b4_4a09_8fe4_70d58f6fc220.slice/crio-ee88c980d3f3713dee5d9e1e87be7852f314ff80982ef61f9425aa997ce5882d WatchSource:0}: Error finding container ee88c980d3f3713dee5d9e1e87be7852f314ff80982ef61f9425aa997ce5882d: Status 404 returned error can't find the container with id ee88c980d3f3713dee5d9e1e87be7852f314ff80982ef61f9425aa997ce5882d
Jul 28 14:53:39.035751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.035709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" event={"ID":"2219b07b-41b4-4a09-8fe4-70d58f6fc220","Type":"ContainerStarted","Data":"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"}
Jul 28 14:53:39.035973 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.035781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" event={"ID":"2219b07b-41b4-4a09-8fe4-70d58f6fc220","Type":"ContainerStarted","Data":"ee88c980d3f3713dee5d9e1e87be7852f314ff80982ef61f9425aa997ce5882d"}
Jul 28 14:53:39.035973 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.035899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:39.037433 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.037407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" event={"ID":"361be0c7-00df-45db-81ef-5cd85e2ff85e","Type":"ContainerStarted","Data":"19dc42ccb1c49f33cb67d7be80e777582d3aef6ccde21c4657c2e4dabd2af850"}
Jul 28 14:53:39.037554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.037454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" event={"ID":"361be0c7-00df-45db-81ef-5cd85e2ff85e","Type":"ContainerStarted","Data":"295c07178c56d3d55242e704ac3a00171802ae3f0aa8ceadb1585d801ce34587"}
Jul 28 14:53:39.037554 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.037479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:39.039260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.039235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" event={"ID":"5dbe4caf-df98-42d4-93ce-ca41eb78c966","Type":"ContainerStarted","Data":"215e49facbe9d130c5d028c5102376d115ce3901d9c352eeddcb7df9c457f336"}
Jul 28 14:53:39.039360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.039283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:39.039360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.039307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" event={"ID":"5dbe4caf-df98-42d4-93ce-ca41eb78c966","Type":"ContainerStarted","Data":"59d85994ad7809a030d44005afb6f5f161e0875213fd8e00e8e66104b4df4f51"}
Jul 28 14:53:39.051815 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.051765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" podStartSLOduration=1.051745275 podStartE2EDuration="1.051745275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:53:39.050582334 +0000 UTC m=+1856.736660446" watchObservedRunningTime="2026-07-28 14:53:39.051745275 +0000 UTC m=+1856.737823366"
Jul 28 14:53:39.067174 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.067107    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" podStartSLOduration=1.64349742 podStartE2EDuration="2.067084983s" totalImagesPullingTime="423.587563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:53:39.064933161 +0000 UTC m=+1856.751011285" watchObservedRunningTime="2026-07-28 14:53:39.067084983 +0000 UTC m=+1856.753163097"
Jul 28 14:53:39.078856 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.078773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" podStartSLOduration=1.078754488 podStartE2EDuration="1.078754488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:53:39.078164276 +0000 UTC m=+1856.764242391" watchObservedRunningTime="2026-07-28 14:53:39.078754488 +0000 UTC m=+1856.764832600"
Jul 28 14:53:39.501932 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.501906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:39.579828 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.579828 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.579828 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84fqs\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579846    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.579984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert\") pod \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\" (UID: \"23d0b470-e3bb-4dec-8b5e-d4fa6f834746\") "
Jul 28 14:53:39.580084 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:53:39.580421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:53:39.580421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580272    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:53:39.580421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580340    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.580421 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580358    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.580622 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.580482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:53:39.581212 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.581178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data" (OuterVolumeSpecName: "istio-data") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:53:39.582001 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.581975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:53:39.582184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.582159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs" (OuterVolumeSpecName: "kube-api-access-84fqs") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "kube-api-access-84fqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:53:39.582184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.582175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token" (OuterVolumeSpecName: "istio-token") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:53:39.582329 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.582314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23d0b470-e3bb-4dec-8b5e-d4fa6f834746" (UID: "23d0b470-e3bb-4dec-8b5e-d4fa6f834746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 14:53:39.681325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681279    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681316    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fqs\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-kube-api-access-84fqs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681327    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681335    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681645 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681344    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681645 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681352    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:39.681645 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:39.681361    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d0b470-e3bb-4dec-8b5e-d4fa6f834746-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:53:40.050568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.050510    2577 generic.go:356] "Generic (PLEG): container finished" podID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerID="24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d" exitCode=0
Jul 28 14:53:40.050769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.050565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerDied","Data":"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"}
Jul 28 14:53:40.050769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.050580    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"
Jul 28 14:53:40.050769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.050628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj" event={"ID":"23d0b470-e3bb-4dec-8b5e-d4fa6f834746","Type":"ContainerDied","Data":"9f152d380fedc359db38559f24f27a8da3046ec6181312a6a2fbc549ea4a53aa"}
Jul 28 14:53:40.050769 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.050660    2577 scope.go:117] "RemoveContainer" containerID="24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"
Jul 28 14:53:40.060377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.060359    2577 scope.go:117] "RemoveContainer" containerID="a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"
Jul 28 14:53:40.068893 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.068848    2577 scope.go:117] "RemoveContainer" containerID="24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"
Jul 28 14:53:40.069693 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:53:40.069206    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d\": container with ID starting with 24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d not found: ID does not exist" containerID="24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"
Jul 28 14:53:40.069693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.069271    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d"} err="failed to get container status \"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d\": rpc error: code = NotFound desc = could not find container \"24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d\": container with ID starting with 24e9942234bcfebe1daa2fd896b1e11ca237ac43c1335e499d628ac9bf0f920d not found: ID does not exist"
Jul 28 14:53:40.069693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.069320    2577 scope.go:117] "RemoveContainer" containerID="a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"
Jul 28 14:53:40.069693 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:53:40.069656    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1\": container with ID starting with a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1 not found: ID does not exist" containerID="a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"
Jul 28 14:53:40.069693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.069681    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1"} err="failed to get container status \"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1\": rpc error: code = NotFound desc = could not find container \"a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1\": container with ID starting with a5b254d99bade0cfeaac377b44d770e028cabfbc9a1025b127bdc6b97d110bc1 not found: ID does not exist"
Jul 28 14:53:40.071791 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.071770    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:53:40.078741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.078718    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-856ad4e0-openshift-default-54b55f5866-wslxj"]
Jul 28 14:53:40.785672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:40.785639    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" path="/var/lib/kubelet/pods/23d0b470-e3bb-4dec-8b5e-d4fa6f834746/volumes"
Jul 28 14:53:45.058592 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:45.058531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:53:50.053389 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:50.053353    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:53:50.053894 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:50.053666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:53:52.320615 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:52.320580    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:55432->10.133.0.42:15021: read: connection reset by peer" start-of-body=
Jul 28 14:53:52.321002 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:52.320650    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:55432->10.133.0.42:15021: read: connection reset by peer"
Jul 28 14:53:53.115145 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.115098    2577 generic.go:356] "Generic (PLEG): container finished" podID="569cf044-9537-4a43-824a-c47042d227ca" containerID="7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e" exitCode=0
Jul 28 14:53:53.115341 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.115155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerDied","Data":"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"}
Jul 28 14:53:53.115646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.115627    2577 scope.go:117] "RemoveContainer" containerID="7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"
Jul 28 14:53:53.116690 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.116659    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:53.116761 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.116745    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:53.116803 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:53.116793    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:53:54.125408 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:54.125362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerStarted","Data":"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"}
Jul 28 14:53:54.732060 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:54.732022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:54.732345 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:54.732317    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 28 14:53:54.732405 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:54.732381    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 28 14:53:55.732039 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:55.731992    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 28 14:53:55.732458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:55.732073    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 28 14:53:56.732784 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:56.732748    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 28 14:53:56.733186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:56.732807    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 28 14:53:57.732591 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:57.732521    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 28 14:53:57.732996 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:57.732619    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 28 14:53:58.736377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:58.736346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:58.736778 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:58.736712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:53:58.737635 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:53:58.737614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:54:08.862519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.862442    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:54:08.863087 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.862676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" podUID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" containerName="maas-api" containerID="cri-o://215e49facbe9d130c5d028c5102376d115ce3901d9c352eeddcb7df9c457f336" gracePeriod=30
Jul 28 14:54:08.916971 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.916938    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:54:08.917215 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.917159    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" podUID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" containerName="payload-processing" containerID="cri-o://e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e" gracePeriod=30
Jul 28 14:54:08.924426 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.924376    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:54:08.924679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:08.924651    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" podUID="361be0c7-00df-45db-81ef-5cd85e2ff85e" containerName="payload-pre-processing" containerID="cri-o://19dc42ccb1c49f33cb67d7be80e777582d3aef6ccde21c4657c2e4dabd2af850" gracePeriod=30
Jul 28 14:54:09.180156 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.179853    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:54:09.183333 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.183298    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:54:09.183705 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.183678    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" containerID="cri-o://b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4" gracePeriod=30
Jul 28 14:54:09.206525 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.206480    2577 generic.go:356] "Generic (PLEG): container finished" podID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" containerID="e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e" exitCode=0
Jul 28 14:54:09.206750 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.206575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" event={"ID":"2219b07b-41b4-4a09-8fe4-70d58f6fc220","Type":"ContainerDied","Data":"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"}
Jul 28 14:54:09.206750 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.206613    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"
Jul 28 14:54:09.206750 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.206665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6" event={"ID":"2219b07b-41b4-4a09-8fe4-70d58f6fc220","Type":"ContainerDied","Data":"ee88c980d3f3713dee5d9e1e87be7852f314ff80982ef61f9425aa997ce5882d"}
Jul 28 14:54:09.206961 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.206753    2577 scope.go:117] "RemoveContainer" containerID="e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"
Jul 28 14:54:09.208486 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.208464    2577 generic.go:356] "Generic (PLEG): container finished" podID="361be0c7-00df-45db-81ef-5cd85e2ff85e" containerID="19dc42ccb1c49f33cb67d7be80e777582d3aef6ccde21c4657c2e4dabd2af850" exitCode=0
Jul 28 14:54:09.208657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.208550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" event={"ID":"361be0c7-00df-45db-81ef-5cd85e2ff85e","Type":"ContainerDied","Data":"19dc42ccb1c49f33cb67d7be80e777582d3aef6ccde21c4657c2e4dabd2af850"}
Jul 28 14:54:09.210342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.210320    2577 generic.go:356] "Generic (PLEG): container finished" podID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" containerID="215e49facbe9d130c5d028c5102376d115ce3901d9c352eeddcb7df9c457f336" exitCode=0
Jul 28 14:54:09.210458 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.210392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" event={"ID":"5dbe4caf-df98-42d4-93ce-ca41eb78c966","Type":"ContainerDied","Data":"215e49facbe9d130c5d028c5102376d115ce3901d9c352eeddcb7df9c457f336"}
Jul 28 14:54:09.218187 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.218168    2577 scope.go:117] "RemoveContainer" containerID="e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"
Jul 28 14:54:09.218526 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:09.218502    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e\": container with ID starting with e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e not found: ID does not exist" containerID="e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"
Jul 28 14:54:09.218652 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.218557    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e"} err="failed to get container status \"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e\": rpc error: code = NotFound desc = could not find container \"e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e\": container with ID starting with e5d96e96b9e22213cf5a6db3c38ab3c99a3f2ad10606066aee41f48a35810c9e not found: ID does not exist"
Jul 28 14:54:09.224239 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.224218    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:54:09.237332 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.237311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:54:09.262389 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.262360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume\") pod \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") "
Jul 28 14:54:09.262589 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.262407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jh96\" (UniqueName: \"kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96\") pod \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\" (UID: \"2219b07b-41b4-4a09-8fe4-70d58f6fc220\") "
Jul 28 14:54:09.262827 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.262800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2219b07b-41b4-4a09-8fe4-70d58f6fc220" (UID: "2219b07b-41b4-4a09-8fe4-70d58f6fc220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:09.264427 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.264404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96" (OuterVolumeSpecName: "kube-api-access-4jh96") pod "2219b07b-41b4-4a09-8fe4-70d58f6fc220" (UID: "2219b07b-41b4-4a09-8fe4-70d58f6fc220"). InnerVolumeSpecName "kube-api-access-4jh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:09.363637 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.363598    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls\") pod \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") "
Jul 28 14:54:09.363843 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.363710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca\") pod \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") "
Jul 28 14:54:09.363843 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.363741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume\") pod \"361be0c7-00df-45db-81ef-5cd85e2ff85e\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") "
Jul 28 14:54:09.363843 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.363772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbh76\" (UniqueName: \"kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76\") pod \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\" (UID: \"5dbe4caf-df98-42d4-93ce-ca41eb78c966\") "
Jul 28 14:54:09.363843 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.363796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4bbc\" (UniqueName: \"kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc\") pod \"361be0c7-00df-45db-81ef-5cd85e2ff85e\" (UID: \"361be0c7-00df-45db-81ef-5cd85e2ff85e\") "
Jul 28 14:54:09.364105 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.364079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dbe4caf-df98-42d4-93ce-ca41eb78c966" (UID: "5dbe4caf-df98-42d4-93ce-ca41eb78c966"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:09.364165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.364107    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2219b07b-41b4-4a09-8fe4-70d58f6fc220-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.364165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.364125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jh96\" (UniqueName: \"kubernetes.io/projected/2219b07b-41b4-4a09-8fe4-70d58f6fc220-kube-api-access-4jh96\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.364271 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.364188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "361be0c7-00df-45db-81ef-5cd85e2ff85e" (UID: "361be0c7-00df-45db-81ef-5cd85e2ff85e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:09.365976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.365940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76" (OuterVolumeSpecName: "kube-api-access-bbh76") pod "5dbe4caf-df98-42d4-93ce-ca41eb78c966" (UID: "5dbe4caf-df98-42d4-93ce-ca41eb78c966"). InnerVolumeSpecName "kube-api-access-bbh76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:09.365976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.365946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc" (OuterVolumeSpecName: "kube-api-access-v4bbc") pod "361be0c7-00df-45db-81ef-5cd85e2ff85e" (UID: "361be0c7-00df-45db-81ef-5cd85e2ff85e"). InnerVolumeSpecName "kube-api-access-v4bbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:09.365976 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.365951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dbe4caf-df98-42d4-93ce-ca41eb78c966" (UID: "5dbe4caf-df98-42d4-93ce-ca41eb78c966"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:54:09.465318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.465287    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbe4caf-df98-42d4-93ce-ca41eb78c966-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.465318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.465317    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/361be0c7-00df-45db-81ef-5cd85e2ff85e-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.465519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.465328    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbh76\" (UniqueName: \"kubernetes.io/projected/5dbe4caf-df98-42d4-93ce-ca41eb78c966-kube-api-access-bbh76\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.465519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.465336    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4bbc\" (UniqueName: \"kubernetes.io/projected/361be0c7-00df-45db-81ef-5cd85e2ff85e-kube-api-access-v4bbc\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.465519 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.465345    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbe4caf-df98-42d4-93ce-ca41eb78c966-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:09.527863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.527829    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:54:09.533102 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.533077    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4033a9f3-57c656c87c-hmjq6"]
Jul 28 14:54:09.999065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999031    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:09.999594 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999576    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999608    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" containerName="maas-api"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999616    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" containerName="maas-api"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999641    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361be0c7-00df-45db-81ef-5cd85e2ff85e" containerName="payload-pre-processing"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="361be0c7-00df-45db-81ef-5cd85e2ff85e" containerName="payload-pre-processing"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999669    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" containerName="payload-processing"
Jul 28 14:54:09.999688 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999677    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" containerName="payload-processing"
Jul 28 14:54:10.000025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999774    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" containerName="payload-processing"
Jul 28 14:54:10.000025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:10.000025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999799    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="361be0c7-00df-45db-81ef-5cd85e2ff85e" containerName="payload-pre-processing"
Jul 28 14:54:10.000025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999809    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:10.000025 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:09.999817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" containerName="maas-api"
Jul 28 14:54:10.002145 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.002124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.004504 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.004482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-38ffd343-openshift-default-dockercfg-q697r\""
Jul 28 14:54:10.014369 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.014342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:10.076793 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.076793 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkxp\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077031 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.076966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.077297 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.077059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178392 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178392 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkxp\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.178657 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.178628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.179062 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.179038    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.179062 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.179058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.179156 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.179134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.179348 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.179328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.179580 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.179562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.181420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.181397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.181640 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.181622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.188912 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.188884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.189034 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.188971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkxp\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp\") pod \"e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") " pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.217291 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.217231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq" event={"ID":"361be0c7-00df-45db-81ef-5cd85e2ff85e","Type":"ContainerDied","Data":"295c07178c56d3d55242e704ac3a00171802ae3f0aa8ceadb1585d801ce34587"}
Jul 28 14:54:10.217291 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.217247    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"
Jul 28 14:54:10.217291 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.217300    2577 scope.go:117] "RemoveContainer" containerID="19dc42ccb1c49f33cb67d7be80e777582d3aef6ccde21c4657c2e4dabd2af850"
Jul 28 14:54:10.218829 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.218809    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"
Jul 28 14:54:10.218914 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.218810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w" event={"ID":"5dbe4caf-df98-42d4-93ce-ca41eb78c966","Type":"ContainerDied","Data":"59d85994ad7809a030d44005afb6f5f161e0875213fd8e00e8e66104b4df4f51"}
Jul 28 14:54:10.226097 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.226072    2577 scope.go:117] "RemoveContainer" containerID="215e49facbe9d130c5d028c5102376d115ce3901d9c352eeddcb7df9c457f336"
Jul 28 14:54:10.248486 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.248454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:54:10.250130 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.250104    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4033a9f3-7b7899f49b-gdjgq"]
Jul 28 14:54:10.259725 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.259689    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:54:10.264564 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.264523    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4033a9f3-6b56dfff6f-mw28w"]
Jul 28 14:54:10.314328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.314281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:10.453158 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.453072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:10.456497 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:10.456465    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be3a23e_d7bd_4bb4_aafc_89b317aa4f3f.slice/crio-73c51b55292fda4e851f5f920bdad866d4242a80e53b23759bedfe57d004b8b5 WatchSource:0}: Error finding container 73c51b55292fda4e851f5f920bdad866d4242a80e53b23759bedfe57d004b8b5: Status 404 returned error can't find the container with id 73c51b55292fda4e851f5f920bdad866d4242a80e53b23759bedfe57d004b8b5
Jul 28 14:54:10.458755 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.458723    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:10.458842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.458793    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:10.458842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.458824    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:10.787474 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.787445    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2219b07b-41b4-4a09-8fe4-70d58f6fc220" path="/var/lib/kubelet/pods/2219b07b-41b4-4a09-8fe4-70d58f6fc220/volumes"
Jul 28 14:54:10.787884 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.787868    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361be0c7-00df-45db-81ef-5cd85e2ff85e" path="/var/lib/kubelet/pods/361be0c7-00df-45db-81ef-5cd85e2ff85e/volumes"
Jul 28 14:54:10.788208 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:10.788197    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbe4caf-df98-42d4-93ce-ca41eb78c966" path="/var/lib/kubelet/pods/5dbe4caf-df98-42d4-93ce-ca41eb78c966/volumes"
Jul 28 14:54:11.226454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:11.226422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerStarted","Data":"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"}
Jul 28 14:54:11.226915 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:11.226474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerStarted","Data":"73c51b55292fda4e851f5f920bdad866d4242a80e53b23759bedfe57d004b8b5"}
Jul 28 14:54:11.245773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:11.245709    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podStartSLOduration=2.245687292 podStartE2EDuration="2.245687292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:11.244165322 +0000 UTC m=+1888.930243448" watchObservedRunningTime="2026-07-28 14:54:11.245687292 +0000 UTC m=+1888.931765405"
Jul 28 14:54:11.314610 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:11.314575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:11.319966 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:11.319938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:12.231774 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.231742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:12.232915 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.232892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:12.671842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.671716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:12.672200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.672179    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:12.672299 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.672203    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d0b470-e3bb-4dec-8b5e-d4fa6f834746" containerName="istio-proxy"
Jul 28 14:54:12.685363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.680409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.685363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.683282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-38ffd343-serving-cert\""
Jul 28 14:54:12.685363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.684993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:12.686363 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.686340    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:54:12.804599 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.804527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.804792 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.804670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.804841 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.804794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brwc\" (UniqueName: \"kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.905629 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.905593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.905835 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.905688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brwc\" (UniqueName: \"kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.905835 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.905733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.906409 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.906385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.908153 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.908127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:12.913977 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.913952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brwc\" (UniqueName: \"kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc\") pod \"maas-api-e2e-ait-38ffd343-686dd99876-m8m9x\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:13.000032 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:12.999993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:13.157917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.157882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:13.159440 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:13.159418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18525fe6_16b9_4563_b24c_e8f0a2209576.slice/crio-2a226eadb3b6a08d8780f7497b6d015f35d93d3bcae9551ae7fb83dbb4fd64f7 WatchSource:0}: Error finding container 2a226eadb3b6a08d8780f7497b6d015f35d93d3bcae9551ae7fb83dbb4fd64f7: Status 404 returned error can't find the container with id 2a226eadb3b6a08d8780f7497b6d015f35d93d3bcae9551ae7fb83dbb4fd64f7
Jul 28 14:54:13.238838 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.238799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" event={"ID":"18525fe6-16b9-4563-b24c-e8f0a2209576","Type":"ContainerStarted","Data":"2a226eadb3b6a08d8780f7497b6d015f35d93d3bcae9551ae7fb83dbb4fd64f7"}
Jul 28 14:54:13.378693 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.378517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:13.381368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.381351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.383616 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.383594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-38ffd343\""
Jul 28 14:54:13.383703 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.383649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-38ffd343-dockercfg-cmwv2\""
Jul 28 14:54:13.391869 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.391840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:13.394697 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.394669    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:13.397308 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.397289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.409486 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.409460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:13.512701 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.512667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.512867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.512719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtxg\" (UniqueName: \"kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.512867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.512775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.512867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.512817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnk2\" (UniqueName: \"kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.613733 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.613694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.613917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.613750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtxg\" (UniqueName: \"kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.613917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.613821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.613917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.613868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnk2\" (UniqueName: \"kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.614610 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.614592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.614673 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.614657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.621905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.621870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtxg\" (UniqueName: \"kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg\") pod \"payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.622007 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.621912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnk2\" (UniqueName: \"kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2\") pod \"payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") " pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.694043 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.694011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:13.708937 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.708906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:13.740360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.740201    2577 patch_prober.go:28] interesting pod/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 14:54:13.740523 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.740416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 14:54:13.841501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.841472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:13.845964 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:13.845920    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8dd5c9_1b49_4e7c_896d_0fc42cf42d75.slice/crio-cf5d3b3058a159c4194bf937c8ecf354c3daf929ef4c06bc0a698dc330c30cb6 WatchSource:0}: Error finding container cf5d3b3058a159c4194bf937c8ecf354c3daf929ef4c06bc0a698dc330c30cb6: Status 404 returned error can't find the container with id cf5d3b3058a159c4194bf937c8ecf354c3daf929ef4c06bc0a698dc330c30cb6
Jul 28 14:54:13.873761 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:13.873735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:13.879701 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:13.879675    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7affe0_9aa9_4461_b104_01852c93efb8.slice/crio-07a277139379767ee74e5b2e5a5170627f4d53ac7d27ed929c8e057eea68dbbf WatchSource:0}: Error finding container 07a277139379767ee74e5b2e5a5170627f4d53ac7d27ed929c8e057eea68dbbf: Status 404 returned error can't find the container with id 07a277139379767ee74e5b2e5a5170627f4d53ac7d27ed929c8e057eea68dbbf
Jul 28 14:54:14.245441 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.245405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" event={"ID":"18525fe6-16b9-4563-b24c-e8f0a2209576","Type":"ContainerStarted","Data":"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"}
Jul 28 14:54:14.245929 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.245572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:14.247287 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.247263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" event={"ID":"6a7affe0-9aa9-4461-b104-01852c93efb8","Type":"ContainerStarted","Data":"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"}
Jul 28 14:54:14.247404 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.247305    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" event={"ID":"6a7affe0-9aa9-4461-b104-01852c93efb8","Type":"ContainerStarted","Data":"07a277139379767ee74e5b2e5a5170627f4d53ac7d27ed929c8e057eea68dbbf"}
Jul 28 14:54:14.247404 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.247334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:14.248905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.248884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" event={"ID":"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75","Type":"ContainerStarted","Data":"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"}
Jul 28 14:54:14.248986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.248929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" event={"ID":"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75","Type":"ContainerStarted","Data":"cf5d3b3058a159c4194bf937c8ecf354c3daf929ef4c06bc0a698dc330c30cb6"}
Jul 28 14:54:14.262717 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.262674    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" podStartSLOduration=1.805274336 podStartE2EDuration="2.262660036s" totalImagesPullingTime="457.3857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:14.260992939 +0000 UTC m=+1891.947071053" watchObservedRunningTime="2026-07-28 14:54:14.262660036 +0000 UTC m=+1891.948738148"
Jul 28 14:54:14.278544 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.278490    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" podStartSLOduration=1.278465722 podStartE2EDuration="1.278465722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:14.276979161 +0000 UTC m=+1891.963057291" watchObservedRunningTime="2026-07-28 14:54:14.278465722 +0000 UTC m=+1891.964543835"
Jul 28 14:54:14.292970 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.292926    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" podStartSLOduration=1.292910955 podStartE2EDuration="1.292910955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:14.290987588 +0000 UTC m=+1891.977065704" watchObservedRunningTime="2026-07-28 14:54:14.292910955 +0000 UTC m=+1891.978989068"
Jul 28 14:54:14.459331 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.459307    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:54:14.625166 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625378 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625176    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625378 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625378 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q26w\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625378 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625286    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625378 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert\") pod \"569cf044-9537-4a43-824a-c47042d227ca\" (UID: \"569cf044-9537-4a43-824a-c47042d227ca\") "
Jul 28 14:54:14.625672 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:14.625879 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.625787    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.626351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.626111    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:14.626351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.626272    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:14.626351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.626329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:14.626604 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.626485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:14.627683 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.627661    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:14.627784 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.627699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:14.627836 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.627803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w" (OuterVolumeSpecName: "kube-api-access-7q26w") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "kube-api-access-7q26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:14.628131 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.628114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "569cf044-9537-4a43-824a-c47042d227ca" (UID: "569cf044-9537-4a43-824a-c47042d227ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726767    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726796    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726806    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726814    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569cf044-9537-4a43-824a-c47042d227ca-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726822    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569cf044-9537-4a43-824a-c47042d227ca-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726830    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726840    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q26w\" (UniqueName: \"kubernetes.io/projected/569cf044-9537-4a43-824a-c47042d227ca-kube-api-access-7q26w\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:14.726855 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:14.726849    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569cf044-9537-4a43-824a-c47042d227ca-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:15.256460 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.256423    2577 generic.go:356] "Generic (PLEG): container finished" podID="569cf044-9537-4a43-824a-c47042d227ca" containerID="b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4" exitCode=0
Jul 28 14:54:15.256955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.256483    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"
Jul 28 14:54:15.256955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.256520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerDied","Data":"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"}
Jul 28 14:54:15.256955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.256584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh" event={"ID":"569cf044-9537-4a43-824a-c47042d227ca","Type":"ContainerDied","Data":"7074de4da91a4fc0181ceba8beaa973fb1c0e2f50458ddd6d632c1e00010f022"}
Jul 28 14:54:15.256955 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.256606    2577 scope.go:117] "RemoveContainer" containerID="b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"
Jul 28 14:54:15.263983 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.257400    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:15.272405 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.272386    2577 scope.go:117] "RemoveContainer" containerID="7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"
Jul 28 14:54:15.277873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.277851    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:54:15.281686 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.281660    2577 scope.go:117] "RemoveContainer" containerID="b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"
Jul 28 14:54:15.281834 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.281813    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4033a9f3-gw-openshift-default-dfb65ddd-ndjmh"]
Jul 28 14:54:15.282008 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:15.281991    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4\": container with ID starting with b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4 not found: ID does not exist" containerID="b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"
Jul 28 14:54:15.282067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.282022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4"} err="failed to get container status \"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4\": rpc error: code = NotFound desc = could not find container \"b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4\": container with ID starting with b9a08fc31e9fbe902ab07541c5dbfa1d6e77670ffe5bab6daf9c0b979bd773b4 not found: ID does not exist"
Jul 28 14:54:15.282067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.282043    2577 scope.go:117] "RemoveContainer" containerID="7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"
Jul 28 14:54:15.282321 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:15.282299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e\": container with ID starting with 7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e not found: ID does not exist" containerID="7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"
Jul 28 14:54:15.282416 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:15.282326    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e"} err="failed to get container status \"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e\": rpc error: code = NotFound desc = could not find container \"7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e\": container with ID starting with 7d58decca74fa8f6579f8a2931fd90b93454200c7944d55891b414915de4a88e not found: ID does not exist"
Jul 28 14:54:16.791302 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:16.791267    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569cf044-9537-4a43-824a-c47042d227ca" path="/var/lib/kubelet/pods/569cf044-9537-4a43-824a-c47042d227ca/volumes"
Jul 28 14:54:20.263711 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:20.263682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:25.259567 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.259514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:25.308963 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.308932    2577 generic.go:356] "Generic (PLEG): container finished" podID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerID="2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f" exitCode=0
Jul 28 14:54:25.309138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.309009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerDied","Data":"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"}
Jul 28 14:54:25.309395 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.309382    2577 scope.go:117] "RemoveContainer" containerID="2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"
Jul 28 14:54:25.310419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.310393    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:25.310523 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.310455    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:25.310523 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:25.310487    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:26.265052 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:26.265021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:26.327585 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:26.327525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerStarted","Data":"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"}
Jul 28 14:54:27.232146 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:27.232104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:27.315272 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:27.315185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:27.317242 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:27.317213    2577 patch_prober.go:28] interesting pod/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 28 14:54:27.317461 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:27.317435    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 28 14:54:28.316968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:28.316931    2577 patch_prober.go:28] interesting pod/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 28 14:54:28.317498 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:28.316991    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 28 14:54:29.315560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:29.315503    2577 patch_prober.go:28] interesting pod/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 28 14:54:29.315872 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:29.315841    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 28 14:54:30.318762 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:30.318731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:30.319725 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:30.319708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:43.475360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.475327    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:43.475975 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.475577    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" podUID="18525fe6-16b9-4563-b24c-e8f0a2209576" containerName="maas-api" containerID="cri-o://6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0" gracePeriod=30
Jul 28 14:54:43.526448 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.526413    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:43.526686 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.526640    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" podUID="6a7affe0-9aa9-4461-b104-01852c93efb8" containerName="payload-processing" containerID="cri-o://1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e" gracePeriod=30
Jul 28 14:54:43.538801 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.538578    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:43.539224 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.539180    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" podUID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" containerName="payload-pre-processing" containerID="cri-o://da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6" gracePeriod=30
Jul 28 14:54:43.813924 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.813887    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:43.829022 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.828969    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:43.833149 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.833123    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:43.911296 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume\") pod \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") "
Jul 28 14:54:43.911296 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnk2\" (UniqueName: \"kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2\") pod \"6a7affe0-9aa9-4461-b104-01852c93efb8\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") "
Jul 28 14:54:43.911597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls\") pod \"18525fe6-16b9-4563-b24c-e8f0a2209576\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") "
Jul 28 14:54:43.911597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume\") pod \"6a7affe0-9aa9-4461-b104-01852c93efb8\" (UID: \"6a7affe0-9aa9-4461-b104-01852c93efb8\") "
Jul 28 14:54:43.911597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgtxg\" (UniqueName: \"kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg\") pod \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\" (UID: \"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75\") "
Jul 28 14:54:43.911597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brwc\" (UniqueName: \"kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc\") pod \"18525fe6-16b9-4563-b24c-e8f0a2209576\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") "
Jul 28 14:54:43.911597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca\") pod \"18525fe6-16b9-4563-b24c-e8f0a2209576\" (UID: \"18525fe6-16b9-4563-b24c-e8f0a2209576\") "
Jul 28 14:54:43.911877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" (UID: "4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:43.911935 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a7affe0-9aa9-4461-b104-01852c93efb8" (UID: "6a7affe0-9aa9-4461-b104-01852c93efb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:43.912021 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.912000    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:43.912080 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.912028    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7affe0-9aa9-4461-b104-01852c93efb8-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:43.912080 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.911997    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18525fe6-16b9-4563-b24c-e8f0a2209576" (UID: "18525fe6-16b9-4563-b24c-e8f0a2209576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:43.913642 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.913614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18525fe6-16b9-4563-b24c-e8f0a2209576" (UID: "18525fe6-16b9-4563-b24c-e8f0a2209576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:54:43.913739 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.913680    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc" (OuterVolumeSpecName: "kube-api-access-9brwc") pod "18525fe6-16b9-4563-b24c-e8f0a2209576" (UID: "18525fe6-16b9-4563-b24c-e8f0a2209576"). InnerVolumeSpecName "kube-api-access-9brwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:43.913739 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.913712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2" (OuterVolumeSpecName: "kube-api-access-jhnk2") pod "6a7affe0-9aa9-4461-b104-01852c93efb8" (UID: "6a7affe0-9aa9-4461-b104-01852c93efb8"). InnerVolumeSpecName "kube-api-access-jhnk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:43.913922 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:43.913889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg" (OuterVolumeSpecName: "kube-api-access-sgtxg") pod "4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" (UID: "4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75"). InnerVolumeSpecName "kube-api-access-sgtxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:44.013150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.013054    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnk2\" (UniqueName: \"kubernetes.io/projected/6a7affe0-9aa9-4461-b104-01852c93efb8-kube-api-access-jhnk2\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:44.013150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.013088    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18525fe6-16b9-4563-b24c-e8f0a2209576-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:44.013150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.013103    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgtxg\" (UniqueName: \"kubernetes.io/projected/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75-kube-api-access-sgtxg\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:44.013150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.013117    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brwc\" (UniqueName: \"kubernetes.io/projected/18525fe6-16b9-4563-b24c-e8f0a2209576-kube-api-access-9brwc\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:44.013150 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.013128    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18525fe6-16b9-4563-b24c-e8f0a2209576-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:44.428454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.428369    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a7affe0-9aa9-4461-b104-01852c93efb8" containerID="1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e" exitCode=0
Jul 28 14:54:44.428454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.428415    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"
Jul 28 14:54:44.428454 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.428447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" event={"ID":"6a7affe0-9aa9-4461-b104-01852c93efb8","Type":"ContainerDied","Data":"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"}
Jul 28 14:54:44.428741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.428499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2" event={"ID":"6a7affe0-9aa9-4461-b104-01852c93efb8","Type":"ContainerDied","Data":"07a277139379767ee74e5b2e5a5170627f4d53ac7d27ed929c8e057eea68dbbf"}
Jul 28 14:54:44.428741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.428525    2577 scope.go:117] "RemoveContainer" containerID="1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"
Jul 28 14:54:44.429822 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.429780    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" containerID="da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6" exitCode=0
Jul 28 14:54:44.429822 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.429806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" event={"ID":"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75","Type":"ContainerDied","Data":"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"}
Jul 28 14:54:44.429941 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.429841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"
Jul 28 14:54:44.430001 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.429848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz" event={"ID":"4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75","Type":"ContainerDied","Data":"cf5d3b3058a159c4194bf937c8ecf354c3daf929ef4c06bc0a698dc330c30cb6"}
Jul 28 14:54:44.431282 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.431260    2577 generic.go:356] "Generic (PLEG): container finished" podID="18525fe6-16b9-4563-b24c-e8f0a2209576" containerID="6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0" exitCode=0
Jul 28 14:54:44.431394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.431300    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"
Jul 28 14:54:44.431394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.431347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" event={"ID":"18525fe6-16b9-4563-b24c-e8f0a2209576","Type":"ContainerDied","Data":"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"}
Jul 28 14:54:44.431394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.431389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x" event={"ID":"18525fe6-16b9-4563-b24c-e8f0a2209576","Type":"ContainerDied","Data":"2a226eadb3b6a08d8780f7497b6d015f35d93d3bcae9551ae7fb83dbb4fd64f7"}
Jul 28 14:54:44.438138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.438117    2577 scope.go:117] "RemoveContainer" containerID="1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"
Jul 28 14:54:44.438456 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:44.438428    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e\": container with ID starting with 1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e not found: ID does not exist" containerID="1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"
Jul 28 14:54:44.438586 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.438466    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e"} err="failed to get container status \"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e\": rpc error: code = NotFound desc = could not find container \"1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e\": container with ID starting with 1693b1d76b64ea908862fc1a806dcecb74fd718c584b8a6490ac3a20c40f380e not found: ID does not exist"
Jul 28 14:54:44.438586 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.438492    2577 scope.go:117] "RemoveContainer" containerID="da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"
Jul 28 14:54:44.447338 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.447324    2577 scope.go:117] "RemoveContainer" containerID="da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"
Jul 28 14:54:44.447604 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:44.447587    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6\": container with ID starting with da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6 not found: ID does not exist" containerID="da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"
Jul 28 14:54:44.447662 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.447611    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6"} err="failed to get container status \"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6\": rpc error: code = NotFound desc = could not find container \"da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6\": container with ID starting with da326cbeebec921fd3d6100c3955b15d633484014698ab1a14bb2e0b20c81da6 not found: ID does not exist"
Jul 28 14:54:44.447662 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.447627    2577 scope.go:117] "RemoveContainer" containerID="6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"
Jul 28 14:54:44.455208 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.455189    2577 scope.go:117] "RemoveContainer" containerID="6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"
Jul 28 14:54:44.455473 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:44.455455    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0\": container with ID starting with 6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0 not found: ID does not exist" containerID="6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"
Jul 28 14:54:44.455618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.455479    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0"} err="failed to get container status \"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0\": rpc error: code = NotFound desc = could not find container \"6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0\": container with ID starting with 6dd583fe3c6b91546108b26b591f0c197b4570a1737f4f26aff71cc4adc55ad0 not found: ID does not exist"
Jul 28 14:54:44.458453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.458434    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:44.462472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.462448    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-38ffd343-69cd497ccd-9zkc2"]
Jul 28 14:54:44.471776 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.471754    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:44.473373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.473355    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-38ffd343-6965bd97d8-c9nnz"]
Jul 28 14:54:44.483368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.483343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:44.487115 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.487094    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-38ffd343-686dd99876-m8m9x"]
Jul 28 14:54:44.786382 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.786350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18525fe6-16b9-4563-b24c-e8f0a2209576" path="/var/lib/kubelet/pods/18525fe6-16b9-4563-b24c-e8f0a2209576/volumes"
Jul 28 14:54:44.786744 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.786730    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" path="/var/lib/kubelet/pods/4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75/volumes"
Jul 28 14:54:44.787037 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:44.787025    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7affe0-9aa9-4461-b104-01852c93efb8" path="/var/lib/kubelet/pods/6a7affe0-9aa9-4461-b104-01852c93efb8/volumes"
Jul 28 14:54:45.025318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.025286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:45.025605 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.025579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy" containerID="cri-o://c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510" gracePeriod=30
Jul 28 14:54:45.321788 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.321744    2577 patch_prober.go:28] interesting pod/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 28 14:54:45.322012 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.321824    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 14:54:45.837155 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837122    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837557    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" containerName="payload-pre-processing"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837573    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" containerName="payload-pre-processing"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837583    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7affe0-9aa9-4461-b104-01852c93efb8" containerName="payload-processing"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7affe0-9aa9-4461-b104-01852c93efb8" containerName="payload-processing"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837604    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837630    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18525fe6-16b9-4563-b24c-e8f0a2209576" containerName="maas-api"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837639    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18525fe6-16b9-4563-b24c-e8f0a2209576" containerName="maas-api"
Jul 28 14:54:45.837646 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837652    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:45.838152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:45.838152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837781    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8dd5c9-1b49-4e7c-896d-0fc42cf42d75" containerName="payload-pre-processing"
Jul 28 14:54:45.838152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837793    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:45.838152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837801    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7affe0-9aa9-4461-b104-01852c93efb8" containerName="payload-processing"
Jul 28 14:54:45.838152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.837806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18525fe6-16b9-4563-b24c-e8f0a2209576" containerName="maas-api"
Jul 28 14:54:45.840768 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.840744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.843173 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.843149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a0bd8cb8-openshift-default-dockercfg-btglr\""
Jul 28 14:54:45.852411 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.852387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:54:45.930312 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7qv\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930671 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:45.930781 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:45.930695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.031959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.031927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.031959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.031963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032185 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.031988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7qv\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032185 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032185 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032430 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032660 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032789 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.032923 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.032819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.034590 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.034566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.034863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.034844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.040272 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.040244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.040514 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.040495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7qv\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv\") pod \"e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") " pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.153459 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.153365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:46.291596 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.291450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:54:46.293953 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:46.293919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5495b031_60fe_49ff_8f84_0f23340b9df3.slice/crio-c4dbaa8dc621e04f105bfcd218097a5310a86e132870872d7e95fd8bc09dab18 WatchSource:0}: Error finding container c4dbaa8dc621e04f105bfcd218097a5310a86e132870872d7e95fd8bc09dab18: Status 404 returned error can't find the container with id c4dbaa8dc621e04f105bfcd218097a5310a86e132870872d7e95fd8bc09dab18
Jul 28 14:54:46.296451 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.296411    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:46.296529 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.296502    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:46.296587 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.296566    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:54:46.445877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.445784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerStarted","Data":"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"}
Jul 28 14:54:46.445877 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.445854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerStarted","Data":"c4dbaa8dc621e04f105bfcd218097a5310a86e132870872d7e95fd8bc09dab18"}
Jul 28 14:54:46.466301 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:46.466211    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podStartSLOduration=1.466188557 podStartE2EDuration="1.466188557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:46.464822904 +0000 UTC m=+1924.150901019" watchObservedRunningTime="2026-07-28 14:54:46.466188557 +0000 UTC m=+1924.152266674"
Jul 28 14:54:47.154418 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.154388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:47.160155 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.160130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:47.450414 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.450330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:47.451335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.451308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:54:47.891786 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.891751    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:54:47.892502 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.892477    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="569cf044-9537-4a43-824a-c47042d227ca" containerName="istio-proxy"
Jul 28 14:54:47.895661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.895641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:47.898124 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.898099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:54:47.898244 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.898128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a0bd8cb8-serving-cert\""
Jul 28 14:54:47.906353 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.906328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:54:47.950453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.950416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:47.950676 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.950519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:47.950676 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:47.950580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8gv\" (UniqueName: \"kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.051917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.051880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.051917 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.051921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8gv\" (UniqueName: \"kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.052183 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.051962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.052513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.052494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.054453 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.054436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.059233 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.059207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8gv\" (UniqueName: \"kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv\") pod \"maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.210466 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.210438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:48.354844 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.354814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:54:48.358085 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:48.358059    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fcb470_6d2e_4b94_bfee_41c8c0b986f3.slice/crio-b03f6e071261c33b4f17fa3cb16288ee9b7c62d495e796ffb5ab4878c069003b WatchSource:0}: Error finding container b03f6e071261c33b4f17fa3cb16288ee9b7c62d495e796ffb5ab4878c069003b: Status 404 returned error can't find the container with id b03f6e071261c33b4f17fa3cb16288ee9b7c62d495e796ffb5ab4878c069003b
Jul 28 14:54:48.456734 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.456699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" event={"ID":"25fcb470-6d2e-4b94-bfee-41c8c0b986f3","Type":"ContainerStarted","Data":"b03f6e071261c33b4f17fa3cb16288ee9b7c62d495e796ffb5ab4878c069003b"}
Jul 28 14:54:48.634788 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.634705    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:54:48.639119 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.639101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.641451 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.641427    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a0bd8cb8-dockercfg-msmtb\""
Jul 28 14:54:48.641601 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.641428    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a0bd8cb8\""
Jul 28 14:54:48.646055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.646029    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:54:48.654605 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.654566    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:54:48.659858 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.659825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.666820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.666793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:54:48.758618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.758593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8kf\" (UniqueName: \"kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.758772 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.758634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.758772 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.758680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.758887 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.758821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgx7\" (UniqueName: \"kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.859935 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.859894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgx7\" (UniqueName: \"kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.860125 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.859945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8kf\" (UniqueName: \"kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.860125 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.859978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.860125 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.860011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.860823 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.860802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.860875 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.860805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.868075 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.868040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8kf\" (UniqueName: \"kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf\") pod \"payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.868203 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.868118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgx7\" (UniqueName: \"kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7\") pod \"payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") " pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:48.952250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.952160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:48.974612 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:48.974581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:49.106684 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.106653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:54:49.107972 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:49.107943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe230dc3_a33c_48e9_90c7_edf660c71394.slice/crio-148a87f56a5bea75c5353859f11c0954fb089005086f7b0fa1dd8a9766fb9125 WatchSource:0}: Error finding container 148a87f56a5bea75c5353859f11c0954fb089005086f7b0fa1dd8a9766fb9125: Status 404 returned error can't find the container with id 148a87f56a5bea75c5353859f11c0954fb089005086f7b0fa1dd8a9766fb9125
Jul 28 14:54:49.140149 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.140001    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:54:49.152938 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:54:49.152898    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acf7625_beff_4529_a5ba_fec094505749.slice/crio-a4f999b8baa5ef79483a556d6c79fe18bf45f0cce9cf3b0f1b685ed03396b822 WatchSource:0}: Error finding container a4f999b8baa5ef79483a556d6c79fe18bf45f0cce9cf3b0f1b685ed03396b822: Status 404 returned error can't find the container with id a4f999b8baa5ef79483a556d6c79fe18bf45f0cce9cf3b0f1b685ed03396b822
Jul 28 14:54:49.463306 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.463273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" event={"ID":"1acf7625-beff-4529-a5ba-fec094505749","Type":"ContainerStarted","Data":"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"}
Jul 28 14:54:49.463805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.463325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" event={"ID":"1acf7625-beff-4529-a5ba-fec094505749","Type":"ContainerStarted","Data":"a4f999b8baa5ef79483a556d6c79fe18bf45f0cce9cf3b0f1b685ed03396b822"}
Jul 28 14:54:49.463805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.463383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:54:49.464908 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.464885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" event={"ID":"be230dc3-a33c-48e9-90c7-edf660c71394","Type":"ContainerStarted","Data":"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"}
Jul 28 14:54:49.465037 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.464925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" event={"ID":"be230dc3-a33c-48e9-90c7-edf660c71394","Type":"ContainerStarted","Data":"148a87f56a5bea75c5353859f11c0954fb089005086f7b0fa1dd8a9766fb9125"}
Jul 28 14:54:49.465037 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.464948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:54:49.466573 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.466549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" event={"ID":"25fcb470-6d2e-4b94-bfee-41c8c0b986f3","Type":"ContainerStarted","Data":"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"}
Jul 28 14:54:49.466715 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.466608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:54:49.479560 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.479481    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" podStartSLOduration=1.479467337 podStartE2EDuration="1.479467337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:49.477825377 +0000 UTC m=+1927.163903506" watchObservedRunningTime="2026-07-28 14:54:49.479467337 +0000 UTC m=+1927.165545450"
Jul 28 14:54:49.494408 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.494356    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" podStartSLOduration=1.494339866 podStartE2EDuration="1.494339866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:49.492025406 +0000 UTC m=+1927.178103512" watchObservedRunningTime="2026-07-28 14:54:49.494339866 +0000 UTC m=+1927.180417979"
Jul 28 14:54:49.510805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:49.510744    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" podStartSLOduration=2.165049568 podStartE2EDuration="2.510721738s" totalImagesPullingTime="345.67217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:54:49.508654743 +0000 UTC m=+1927.194732857" watchObservedRunningTime="2026-07-28 14:54:49.510721738 +0000 UTC m=+1927.196799852"
Jul 28 14:54:50.284490 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.284466    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:50.374325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374325 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhkxp\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:50.374631 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374589    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy\") pod \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\" (UID: \"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f\") "
Jul 28 14:54:50.374996 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:50.374996 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:54:50.374996 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:50.375160 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.374998    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.375160 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.375017    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.375160 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.375030    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.375160 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.375044    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.375366 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.375196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:50.376710 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.376688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:50.376804 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.376721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp" (OuterVolumeSpecName: "kube-api-access-zhkxp") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "kube-api-access-zhkxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:54:50.376989 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.376962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:54:50.377098 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.376964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" (UID: "5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 14:54:50.474147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.474116    2577 generic.go:356] "Generic (PLEG): container finished" podID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerID="c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510" exitCode=0
Jul 28 14:54:50.474588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.474164    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"
Jul 28 14:54:50.474588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.474219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerDied","Data":"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"}
Jul 28 14:54:50.474588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.474273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m" event={"ID":"5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f","Type":"ContainerDied","Data":"73c51b55292fda4e851f5f920bdad866d4242a80e53b23759bedfe57d004b8b5"}
Jul 28 14:54:50.474588 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.474301    2577 scope.go:117] "RemoveContainer" containerID="c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"
Jul 28 14:54:50.475513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.475457    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.475513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.475476    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.475513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.475488    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkxp\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-kube-api-access-zhkxp\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.475513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.475501    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.475513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.475513    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:54:50.484205 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.484185    2577 scope.go:117] "RemoveContainer" containerID="2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"
Jul 28 14:54:50.492440 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.492421    2577 scope.go:117] "RemoveContainer" containerID="c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"
Jul 28 14:54:50.492783 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:50.492758    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510\": container with ID starting with c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510 not found: ID does not exist" containerID="c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"
Jul 28 14:54:50.492923 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.492790    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510"} err="failed to get container status \"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510\": rpc error: code = NotFound desc = could not find container \"c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510\": container with ID starting with c0ce0b52c11713b97ec52cd6b681920322023ba63b1f530b46fe9cd768b44510 not found: ID does not exist"
Jul 28 14:54:50.492923 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.492811    2577 scope.go:117] "RemoveContainer" containerID="2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"
Jul 28 14:54:50.493165 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:54:50.493139    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f\": container with ID starting with 2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f not found: ID does not exist" containerID="2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"
Jul 28 14:54:50.493250 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.493170    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f"} err="failed to get container status \"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f\": rpc error: code = NotFound desc = could not find container \"2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f\": container with ID starting with 2f3e7224c211683c854fe98888c418c6208dce90e47163fc29d569649030d77f not found: ID does not exist"
Jul 28 14:54:50.495760 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.495737    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:50.499138 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.499116    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-38ffd343-openshift-default-d4c68b7b6-gcx8m"]
Jul 28 14:54:50.787608 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:50.787576    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" path="/var/lib/kubelet/pods/5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f/volumes"
Jul 28 14:54:55.480670 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:54:55.480635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:55:00.476667 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:00.476631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:55:00.477117 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:00.476725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:55:02.451932 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.451894    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 28 14:55:02.452351 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.451946    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 28 14:55:02.549437 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.549406    2577 generic.go:356] "Generic (PLEG): container finished" podID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerID="1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f" exitCode=0
Jul 28 14:55:02.549632 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.549478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerDied","Data":"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"}
Jul 28 14:55:02.549953 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.549939    2577 scope.go:117] "RemoveContainer" containerID="1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"
Jul 28 14:55:02.550988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.550965    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:55:02.551065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.551029    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:55:02.551065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:02.551062    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:55:03.555691 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:03.555657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerStarted","Data":"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"}
Jul 28 14:55:04.154188 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:04.154145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:04.154482 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:04.154455    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 28 14:55:04.154558 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:04.154512    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 28 14:55:05.153886 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:05.153854    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 28 14:55:05.154286 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:05.153929    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 28 14:55:06.154394 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:06.154355    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 28 14:55:06.154821 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:06.154408    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 28 14:55:07.153727 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:07.153692    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 28 14:55:07.153902 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:07.153762    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 28 14:55:08.158177 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:08.158148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:08.158621 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:08.158413    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:08.159066 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:08.159050    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:18.645986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.645944    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:55:18.646593 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.646158    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" podUID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" containerName="maas-api" containerID="cri-o://ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37" gracePeriod=30
Jul 28 14:55:18.706928 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.706893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:55:18.707134 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.707107    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" podUID="1acf7625-beff-4529-a5ba-fec094505749" containerName="payload-processing" containerID="cri-o://33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244" gracePeriod=30
Jul 28 14:55:18.713333 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.713296    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:55:18.713568 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.713525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" podUID="be230dc3-a33c-48e9-90c7-edf660c71394" containerName="payload-pre-processing" containerID="cri-o://13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6" gracePeriod=30
Jul 28 14:55:18.948503 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.948464    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:55:18.948756 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.948727    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" containerID="cri-o://2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e" gracePeriod=30
Jul 28 14:55:18.993184 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:18.993143    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:55:19.007812 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.007778    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:55:19.020420 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.020393    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:55:19.053604 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume\") pod \"1acf7625-beff-4529-a5ba-fec094505749\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") "
Jul 28 14:55:19.053779 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053690    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcgx7\" (UniqueName: \"kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7\") pod \"1acf7625-beff-4529-a5ba-fec094505749\" (UID: \"1acf7625-beff-4529-a5ba-fec094505749\") "
Jul 28 14:55:19.053779 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls\") pod \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") "
Jul 28 14:55:19.053779 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053764    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz8gv\" (UniqueName: \"kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv\") pod \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") "
Jul 28 14:55:19.053941 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca\") pod \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\" (UID: \"25fcb470-6d2e-4b94-bfee-41c8c0b986f3\") "
Jul 28 14:55:19.053941 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf8kf\" (UniqueName: \"kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf\") pod \"be230dc3-a33c-48e9-90c7-edf660c71394\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") "
Jul 28 14:55:19.053941 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume\") pod \"be230dc3-a33c-48e9-90c7-edf660c71394\" (UID: \"be230dc3-a33c-48e9-90c7-edf660c71394\") "
Jul 28 14:55:19.054107 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.053985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1acf7625-beff-4529-a5ba-fec094505749" (UID: "1acf7625-beff-4529-a5ba-fec094505749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:55:19.054229 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.054204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25fcb470-6d2e-4b94-bfee-41c8c0b986f3" (UID: "25fcb470-6d2e-4b94-bfee-41c8c0b986f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:55:19.054414 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.054391    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.056200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.054418    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acf7625-beff-4529-a5ba-fec094505749-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.056200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.054477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be230dc3-a33c-48e9-90c7-edf660c71394" (UID: "be230dc3-a33c-48e9-90c7-edf660c71394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:55:19.056466 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.056442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25fcb470-6d2e-4b94-bfee-41c8c0b986f3" (UID: "25fcb470-6d2e-4b94-bfee-41c8c0b986f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:55:19.056565 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.056521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv" (OuterVolumeSpecName: "kube-api-access-tz8gv") pod "25fcb470-6d2e-4b94-bfee-41c8c0b986f3" (UID: "25fcb470-6d2e-4b94-bfee-41c8c0b986f3"). InnerVolumeSpecName "kube-api-access-tz8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:55:19.056638 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.056573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7" (OuterVolumeSpecName: "kube-api-access-gcgx7") pod "1acf7625-beff-4529-a5ba-fec094505749" (UID: "1acf7625-beff-4529-a5ba-fec094505749"). InnerVolumeSpecName "kube-api-access-gcgx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:55:19.056709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.056692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf" (OuterVolumeSpecName: "kube-api-access-bf8kf") pod "be230dc3-a33c-48e9-90c7-edf660c71394" (UID: "be230dc3-a33c-48e9-90c7-edf660c71394"). InnerVolumeSpecName "kube-api-access-bf8kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:55:19.155096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.155012    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.155096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.155045    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8gv\" (UniqueName: \"kubernetes.io/projected/25fcb470-6d2e-4b94-bfee-41c8c0b986f3-kube-api-access-tz8gv\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.155096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.155056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8kf\" (UniqueName: \"kubernetes.io/projected/be230dc3-a33c-48e9-90c7-edf660c71394-kube-api-access-bf8kf\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.155096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.155065    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be230dc3-a33c-48e9-90c7-edf660c71394-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.155096 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.155074    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcgx7\" (UniqueName: \"kubernetes.io/projected/1acf7625-beff-4529-a5ba-fec094505749-kube-api-access-gcgx7\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:19.641589 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.641553    2577 generic.go:356] "Generic (PLEG): container finished" podID="1acf7625-beff-4529-a5ba-fec094505749" containerID="33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244" exitCode=0
Jul 28 14:55:19.641589 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.641584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" event={"ID":"1acf7625-beff-4529-a5ba-fec094505749","Type":"ContainerDied","Data":"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"}
Jul 28 14:55:19.641847 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.641610    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"
Jul 28 14:55:19.641847 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.641633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j" event={"ID":"1acf7625-beff-4529-a5ba-fec094505749","Type":"ContainerDied","Data":"a4f999b8baa5ef79483a556d6c79fe18bf45f0cce9cf3b0f1b685ed03396b822"}
Jul 28 14:55:19.641847 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.641673    2577 scope.go:117] "RemoveContainer" containerID="33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"
Jul 28 14:55:19.643133 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.643114    2577 generic.go:356] "Generic (PLEG): container finished" podID="be230dc3-a33c-48e9-90c7-edf660c71394" containerID="13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6" exitCode=0
Jul 28 14:55:19.643244 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.643155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"
Jul 28 14:55:19.643244 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.643182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" event={"ID":"be230dc3-a33c-48e9-90c7-edf660c71394","Type":"ContainerDied","Data":"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"}
Jul 28 14:55:19.643244 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.643217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926" event={"ID":"be230dc3-a33c-48e9-90c7-edf660c71394","Type":"ContainerDied","Data":"148a87f56a5bea75c5353859f11c0954fb089005086f7b0fa1dd8a9766fb9125"}
Jul 28 14:55:19.644743 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.644720    2577 generic.go:356] "Generic (PLEG): container finished" podID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" containerID="ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37" exitCode=0
Jul 28 14:55:19.644873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.644761    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"
Jul 28 14:55:19.644873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.644781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" event={"ID":"25fcb470-6d2e-4b94-bfee-41c8c0b986f3","Type":"ContainerDied","Data":"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"}
Jul 28 14:55:19.644873 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.644815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x" event={"ID":"25fcb470-6d2e-4b94-bfee-41c8c0b986f3","Type":"ContainerDied","Data":"b03f6e071261c33b4f17fa3cb16288ee9b7c62d495e796ffb5ab4878c069003b"}
Jul 28 14:55:19.651467 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.651257    2577 scope.go:117] "RemoveContainer" containerID="33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"
Jul 28 14:55:19.651790 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:55:19.651598    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244\": container with ID starting with 33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244 not found: ID does not exist" containerID="33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"
Jul 28 14:55:19.651790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.651631    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244"} err="failed to get container status \"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244\": rpc error: code = NotFound desc = could not find container \"33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244\": container with ID starting with 33dced772fa28a660a0fa4f38124fa128a160e74df445180b18deba7540ba244 not found: ID does not exist"
Jul 28 14:55:19.651790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.651656    2577 scope.go:117] "RemoveContainer" containerID="13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"
Jul 28 14:55:19.673412 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.673392    2577 scope.go:117] "RemoveContainer" containerID="13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"
Jul 28 14:55:19.673753 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:55:19.673728    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6\": container with ID starting with 13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6 not found: ID does not exist" containerID="13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"
Jul 28 14:55:19.673859 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.673762    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6"} err="failed to get container status \"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6\": rpc error: code = NotFound desc = could not find container \"13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6\": container with ID starting with 13f981e218a24a6e3e77e5edbead6f42d61f65ed346f435016ea4b70ca6f16e6 not found: ID does not exist"
Jul 28 14:55:19.673859 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.673781    2577 scope.go:117] "RemoveContainer" containerID="ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"
Jul 28 14:55:19.682721 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.682699    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:55:19.682777 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.682727    2577 scope.go:117] "RemoveContainer" containerID="ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"
Jul 28 14:55:19.683095 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:55:19.683070    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37\": container with ID starting with ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37 not found: ID does not exist" containerID="ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"
Jul 28 14:55:19.683218 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.683104    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37"} err="failed to get container status \"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37\": rpc error: code = NotFound desc = could not find container \"ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37\": container with ID starting with ea9e3f6f1beafb601d7aa9a021e7c8f0aa1b4a8139c08f7c39f8a114289eeb37 not found: ID does not exist"
Jul 28 14:55:19.688012 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.687982    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a0bd8cb8-5fc694645d-p787j"]
Jul 28 14:55:19.696772 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.696744    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:55:19.701213 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.701187    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a0bd8cb8-698695b564-7j926"]
Jul 28 14:55:19.713609 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.713576    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:55:19.716565 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:19.716518    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a0bd8cb8-58695cd476-44s4x"]
Jul 28 14:55:20.785598 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:20.785569    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acf7625-beff-4529-a5ba-fec094505749" path="/var/lib/kubelet/pods/1acf7625-beff-4529-a5ba-fec094505749/volumes"
Jul 28 14:55:20.785983 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:20.785961    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" path="/var/lib/kubelet/pods/25fcb470-6d2e-4b94-bfee-41c8c0b986f3/volumes"
Jul 28 14:55:20.786273 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:20.786260    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be230dc3-a33c-48e9-90c7-edf660c71394" path="/var/lib/kubelet/pods/be230dc3-a33c-48e9-90c7-edf660c71394/volumes"
Jul 28 14:55:23.160986 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:23.160946    2577 patch_prober.go:28] interesting pod/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 28 14:55:23.161456 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:23.161007    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 14:55:24.235466 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.235440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:24.307587 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk7qv\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.307824 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.308128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.308128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs\") pod \"5495b031-60fe-49ff-8f84-0f23340b9df3\" (UID: \"5495b031-60fe-49ff-8f84-0f23340b9df3\") "
Jul 28 14:55:24.308128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.307987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:55:24.308128 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:55:24.308335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308267    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.308335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308296    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.308335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:55:24.308643 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308617    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:55:24.308867 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.308842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data" (OuterVolumeSpecName: "istio-data") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:55:24.309944 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.309920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv" (OuterVolumeSpecName: "kube-api-access-mk7qv") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "kube-api-access-mk7qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:55:24.310050 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.310009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:55:24.310050 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.310030    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:55:24.310147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.310132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5495b031-60fe-49ff-8f84-0f23340b9df3" (UID: "5495b031-60fe-49ff-8f84-0f23340b9df3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 14:55:24.408952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408916    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.408952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408945    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5495b031-60fe-49ff-8f84-0f23340b9df3-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.408952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408954    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.408952 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408962    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.409215 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408970    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7qv\" (UniqueName: \"kubernetes.io/projected/5495b031-60fe-49ff-8f84-0f23340b9df3-kube-api-access-mk7qv\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.409215 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408980    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.409215 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.408987    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5495b031-60fe-49ff-8f84-0f23340b9df3-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:55:24.672473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.672385    2577 generic.go:356] "Generic (PLEG): container finished" podID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerID="2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e" exitCode=0
Jul 28 14:55:24.672473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.672428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerDied","Data":"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"}
Jul 28 14:55:24.672473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.672437    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"
Jul 28 14:55:24.672731 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.672482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms" event={"ID":"5495b031-60fe-49ff-8f84-0f23340b9df3","Type":"ContainerDied","Data":"c4dbaa8dc621e04f105bfcd218097a5310a86e132870872d7e95fd8bc09dab18"}
Jul 28 14:55:24.672731 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.672514    2577 scope.go:117] "RemoveContainer" containerID="2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"
Jul 28 14:55:24.681818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.681797    2577 scope.go:117] "RemoveContainer" containerID="1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"
Jul 28 14:55:24.689687 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.689667    2577 scope.go:117] "RemoveContainer" containerID="2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"
Jul 28 14:55:24.689955 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:55:24.689926    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e\": container with ID starting with 2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e not found: ID does not exist" containerID="2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"
Jul 28 14:55:24.690081 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.689967    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e"} err="failed to get container status \"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e\": rpc error: code = NotFound desc = could not find container \"2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e\": container with ID starting with 2ea6bc0b7a5866ce3f803838f9d497c4935d42bada00497c9b94977b0ca2365e not found: ID does not exist"
Jul 28 14:55:24.690081 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.689992    2577 scope.go:117] "RemoveContainer" containerID="1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"
Jul 28 14:55:24.690252 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:55:24.690228    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f\": container with ID starting with 1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f not found: ID does not exist" containerID="1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"
Jul 28 14:55:24.690311 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.690261    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f"} err="failed to get container status \"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f\": rpc error: code = NotFound desc = could not find container \"1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f\": container with ID starting with 1109d4f7bef26ed8abd59642d1c28f5f383691448c514c8057abaa44ccc0ad1f not found: ID does not exist"
Jul 28 14:55:24.692920 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.692898    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:55:24.702512 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.702489    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a0bd8cb8-openshift-default-687fd4f6dd-qvmms"]
Jul 28 14:55:24.785478 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:55:24.785443    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" path="/var/lib/kubelet/pods/5495b031-60fe-49ff-8f84-0f23340b9df3/volumes"
Jul 28 14:57:02.198983 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.198944    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199416    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199430    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199441    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199454    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be230dc3-a33c-48e9-90c7-edf660c71394" containerName="payload-pre-processing"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be230dc3-a33c-48e9-90c7-edf660c71394" containerName="payload-pre-processing"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199466    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acf7625-beff-4529-a5ba-fec094505749" containerName="payload-processing"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199471    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acf7625-beff-4529-a5ba-fec094505749" containerName="payload-processing"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199484    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" containerName="maas-api"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" containerName="maas-api"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199498    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199507 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199503    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199579    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199588    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199593    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be230dc3-a33c-48e9-90c7-edf660c71394" containerName="payload-pre-processing"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199606    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5495b031-60fe-49ff-8f84-0f23340b9df3" containerName="istio-proxy"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199612    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acf7625-beff-4529-a5ba-fec094505749" containerName="payload-processing"
Jul 28 14:57:02.199959 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.199618    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fcb470-6d2e-4b94-bfee-41c8c0b986f3" containerName="maas-api"
Jul 28 14:57:02.202542 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.202514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.204860 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.204831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-efb491e3-openshift-default-dockercfg-d4xvv\""
Jul 28 14:57:02.204860 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.204852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 14:57:02.220102 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.220072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:57:02.300263 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300263 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ctg\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.300515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.300510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401641 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ctg\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.401857 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.401947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402175 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.402084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402308 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.402267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402364 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.402341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402484 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.402395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.402602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.402582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.404237 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.404218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.404501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.404482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.409212 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.409182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.409310 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.409251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ctg\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg\") pod \"e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") " pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.513915 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.513879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:02.665691 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.665660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:57:02.667807 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:57:02.667723    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fc71b4_73bd_45f5_9c4b_4f678b059496.slice/crio-22b74f6fb6da7a68d74848ac92b6f308e3a345a303c3d3f18a8d3b420ec9025b WatchSource:0}: Error finding container 22b74f6fb6da7a68d74848ac92b6f308e3a345a303c3d3f18a8d3b420ec9025b: Status 404 returned error can't find the container with id 22b74f6fb6da7a68d74848ac92b6f308e3a345a303c3d3f18a8d3b420ec9025b
Jul 28 14:57:02.670414 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.670383    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:02.670518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.670469    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:02.670518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:02.670513    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:03.120719 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:03.120686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerStarted","Data":"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"}
Jul 28 14:57:03.120905 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:03.120736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerStarted","Data":"22b74f6fb6da7a68d74848ac92b6f308e3a345a303c3d3f18a8d3b420ec9025b"}
Jul 28 14:57:03.161260 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:03.161191    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podStartSLOduration=1.161173914 podStartE2EDuration="1.161173914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:57:03.160464557 +0000 UTC m=+2060.846542670" watchObservedRunningTime="2026-07-28 14:57:03.161173914 +0000 UTC m=+2060.847252027"
Jul 28 14:57:03.514684 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:03.514645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:03.520207 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:03.520175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:04.126408 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:04.126374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:04.127487 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:04.127469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:05.208321 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.208287    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:05.208922 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.208901    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:05.208972 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.208927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be3a23e-d7bd-4bb4-aafc-89b317aa4f3f" containerName="istio-proxy"
Jul 28 14:57:05.212269 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.212248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.214575 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.214549    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-efb491e3-serving-cert\""
Jul 28 14:57:05.214710 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.214573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:57:05.222471 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.222448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:05.329186 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.329146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j629b\" (UniqueName: \"kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.329501 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.329472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.329692 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.329665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.430526 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.430488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.430773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.430612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j629b\" (UniqueName: \"kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.430773 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.430650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.430773 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:57:05.430671    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-serving-cert: secret "maas-api-e2e-mt-efb491e3-serving-cert" not found
Jul 28 14:57:05.430773 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:57:05.430744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls podName:b5985db3-a7a1-4442-aa31-427e46c8552e nodeName:}" failed. No retries permitted until 2026-07-28 14:57:05.930722902 +0000 UTC m=+2063.616800998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls") pod "maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" (UID: "b5985db3-a7a1-4442-aa31-427e46c8552e") : secret "maas-api-e2e-mt-efb491e3-serving-cert" not found
Jul 28 14:57:05.431335 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.431311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.438745 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.438719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j629b\" (UniqueName: \"kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.818207 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.818172    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:05.821717 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.821695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:05.824017 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.823999    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-efb491e3\""
Jul 28 14:57:05.832266 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.832236    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:05.861561 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.861511    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:05.865063 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.865043    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:05.867067 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.867048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-efb491e3-dockercfg-skbxh\""
Jul 28 14:57:05.877727 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.877704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:05.935524 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.935494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4wm\" (UniqueName: \"kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:05.935751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.935672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:05.935751 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.935711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:05.938133 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:05.938109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") pod \"maas-api-e2e-mt-efb491e3-c9b559b49-6txtd\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:06.037001 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.036960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.037198 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.037024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4wm\" (UniqueName: \"kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:06.037198 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.037128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnsk\" (UniqueName: \"kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.037298 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.037279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:06.038013 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.037994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:06.049108 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.049081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4wm\" (UniqueName: \"kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm\") pod \"payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:06.126606 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.126516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:06.132964 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.132943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:06.137889 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.137860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.138028 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.137978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnsk\" (UniqueName: \"kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.138517 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.138498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.146561 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.146517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnsk\" (UniqueName: \"kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk\") pod \"payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") " pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.175497 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.175441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:06.297047 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.296967    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:06.322758 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.322720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:06.328008 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:57:06.327971    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07373aed_2c05_476b_b5b7_4bf8f3325706.slice/crio-6280555ce8e6800e61e8fbb4c624ec7a00f7e037a969a1c98644b82377177d18 WatchSource:0}: Error finding container 6280555ce8e6800e61e8fbb4c624ec7a00f7e037a969a1c98644b82377177d18: Status 404 returned error can't find the container with id 6280555ce8e6800e61e8fbb4c624ec7a00f7e037a969a1c98644b82377177d18
Jul 28 14:57:06.355028 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:06.355000    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:06.373879 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:57:06.373845    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7b78a7_3e08_45cb_8095_eb918955d8f9.slice/crio-60523c8bce0d8c421907b5e783874a3f2178c80b0c630432e5e44307975fddd6 WatchSource:0}: Error finding container 60523c8bce0d8c421907b5e783874a3f2178c80b0c630432e5e44307975fddd6: Status 404 returned error can't find the container with id 60523c8bce0d8c421907b5e783874a3f2178c80b0c630432e5e44307975fddd6
Jul 28 14:57:07.145691 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.145582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" event={"ID":"2d7b78a7-3e08-45cb-8095-eb918955d8f9","Type":"ContainerStarted","Data":"2cf7e49fe442ca93383dc5f829194e3e83130edc49e26bc20be1f8aae051db4b"}
Jul 28 14:57:07.145691 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.145654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" event={"ID":"2d7b78a7-3e08-45cb-8095-eb918955d8f9","Type":"ContainerStarted","Data":"60523c8bce0d8c421907b5e783874a3f2178c80b0c630432e5e44307975fddd6"}
Jul 28 14:57:07.146328 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.146308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:07.148318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.147694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" event={"ID":"b5985db3-a7a1-4442-aa31-427e46c8552e","Type":"ContainerStarted","Data":"6453174c4d42b77125665777d661209e1dffc85fd554f44bebd9a8e0c59c4e7d"}
Jul 28 14:57:07.148318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.147743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" event={"ID":"b5985db3-a7a1-4442-aa31-427e46c8552e","Type":"ContainerStarted","Data":"8e74d44cb8646e8b9173759d335e12d170990a54511df310db149ee328d25606"}
Jul 28 14:57:07.148318 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.148269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:07.150598 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.150574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" event={"ID":"07373aed-2c05-476b-b5b7-4bf8f3325706","Type":"ContainerStarted","Data":"dc66eb40a9e2f83eb6d074e64948d7dc6ce9e7c69d0d68f79c92943b5e299667"}
Jul 28 14:57:07.150783 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.150764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" event={"ID":"07373aed-2c05-476b-b5b7-4bf8f3325706","Type":"ContainerStarted","Data":"6280555ce8e6800e61e8fbb4c624ec7a00f7e037a969a1c98644b82377177d18"}
Jul 28 14:57:07.150988 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.150968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:07.162192 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.162148    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" podStartSLOduration=2.162132606 podStartE2EDuration="2.162132606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:57:07.16041831 +0000 UTC m=+2064.846496427" watchObservedRunningTime="2026-07-28 14:57:07.162132606 +0000 UTC m=+2064.848210716"
Jul 28 14:57:07.175788 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.175740    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" podStartSLOduration=2.175724768 podStartE2EDuration="2.175724768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:57:07.173875984 +0000 UTC m=+2064.859954098" watchObservedRunningTime="2026-07-28 14:57:07.175724768 +0000 UTC m=+2064.861802881"
Jul 28 14:57:07.191197 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:07.191150    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" podStartSLOduration=1.6414959329999999 podStartE2EDuration="2.191136335s" totalImagesPullingTime="549.640402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:57:07.190449735 +0000 UTC m=+2064.876527849" watchObservedRunningTime="2026-07-28 14:57:07.191136335 +0000 UTC m=+2064.877214448"
Jul 28 14:57:14.167017 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:14.166988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:18.159051 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:18.159024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:19.130055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.129317    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:39860->10.133.0.54:15021: read: connection reset by peer" start-of-body=
Jul 28 14:57:19.130055 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.129394    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:39860->10.133.0.54:15021: read: connection reset by peer"
Jul 28 14:57:19.164283 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.164253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:19.214728 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.214697    2577 generic.go:356] "Generic (PLEG): container finished" podID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerID="54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a" exitCode=0
Jul 28 14:57:19.214900 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.214776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerDied","Data":"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"}
Jul 28 14:57:19.215234 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.215218    2577 scope.go:117] "RemoveContainer" containerID="54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"
Jul 28 14:57:19.216384 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.216352    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:19.216472 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.216437    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:19.223620 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:19.217003    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:57:20.221070 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:20.221030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerStarted","Data":"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"}
Jul 28 14:57:20.514743 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:20.514578    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:20.514743 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:20.514637    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 28 14:57:20.514743 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:20.514697    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 28 14:57:21.515298 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:21.515250    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 28 14:57:21.515841 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:21.515335    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 28 14:57:22.514978 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:22.514942    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 28 14:57:22.515293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:22.515264    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 28 14:57:23.514886 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:23.514843    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 28 14:57:23.515566 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:23.515505    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 28 14:57:24.514713 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:24.514676    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 28 14:57:24.514898 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:24.514732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 28 14:57:25.518015 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:25.517983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:25.518473 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:25.518316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:25.519066 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:25.519047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:57:42.861818 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:42.861782    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:57:42.865479 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:42.865457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:57:42.868342 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:42.868321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 14:57:42.871655 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:42.871638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 14:57:48.145545 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.145504    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:48.145928 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.145781    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" podUID="b5985db3-a7a1-4442-aa31-427e46c8552e" containerName="maas-api" containerID="cri-o://6453174c4d42b77125665777d661209e1dffc85fd554f44bebd9a8e0c59c4e7d" gracePeriod=30
Jul 28 14:57:48.192531 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.192499    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:48.192805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.192760    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" podUID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" containerName="payload-processing" containerID="cri-o://2cf7e49fe442ca93383dc5f829194e3e83130edc49e26bc20be1f8aae051db4b" gracePeriod=30
Jul 28 14:57:48.199906 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.199877    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:48.200124 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.200090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" podUID="07373aed-2c05-476b-b5b7-4bf8f3325706" containerName="payload-pre-processing" containerID="cri-o://dc66eb40a9e2f83eb6d074e64948d7dc6ce9e7c69d0d68f79c92943b5e299667" gracePeriod=30
Jul 28 14:57:48.365360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.365126    2577 generic.go:356] "Generic (PLEG): container finished" podID="07373aed-2c05-476b-b5b7-4bf8f3325706" containerID="dc66eb40a9e2f83eb6d074e64948d7dc6ce9e7c69d0d68f79c92943b5e299667" exitCode=0
Jul 28 14:57:48.365360 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.365296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" event={"ID":"07373aed-2c05-476b-b5b7-4bf8f3325706","Type":"ContainerDied","Data":"dc66eb40a9e2f83eb6d074e64948d7dc6ce9e7c69d0d68f79c92943b5e299667"}
Jul 28 14:57:48.370152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.367881    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" containerID="2cf7e49fe442ca93383dc5f829194e3e83130edc49e26bc20be1f8aae051db4b" exitCode=0
Jul 28 14:57:48.370152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.367986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" event={"ID":"2d7b78a7-3e08-45cb-8095-eb918955d8f9","Type":"ContainerDied","Data":"2cf7e49fe442ca93383dc5f829194e3e83130edc49e26bc20be1f8aae051db4b"}
Jul 28 14:57:48.370152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.369901    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5985db3-a7a1-4442-aa31-427e46c8552e" containerID="6453174c4d42b77125665777d661209e1dffc85fd554f44bebd9a8e0c59c4e7d" exitCode=0
Jul 28 14:57:48.370152 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.369988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" event={"ID":"b5985db3-a7a1-4442-aa31-427e46c8552e","Type":"ContainerDied","Data":"6453174c4d42b77125665777d661209e1dffc85fd554f44bebd9a8e0c59c4e7d"}
Jul 28 14:57:48.462699 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.462671    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:48.483182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.483150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") pod \"b5985db3-a7a1-4442-aa31-427e46c8552e\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") "
Jul 28 14:57:48.483365 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.483223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca\") pod \"b5985db3-a7a1-4442-aa31-427e46c8552e\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") "
Jul 28 14:57:48.483466 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.483438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j629b\" (UniqueName: \"kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b\") pod \"b5985db3-a7a1-4442-aa31-427e46c8552e\" (UID: \"b5985db3-a7a1-4442-aa31-427e46c8552e\") "
Jul 28 14:57:48.483815 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.483785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5985db3-a7a1-4442-aa31-427e46c8552e" (UID: "b5985db3-a7a1-4442-aa31-427e46c8552e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:57:48.485521 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.485493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5985db3-a7a1-4442-aa31-427e46c8552e" (UID: "b5985db3-a7a1-4442-aa31-427e46c8552e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:57:48.485999 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.485973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b" (OuterVolumeSpecName: "kube-api-access-j629b") pod "b5985db3-a7a1-4442-aa31-427e46c8552e" (UID: "b5985db3-a7a1-4442-aa31-427e46c8552e"). InnerVolumeSpecName "kube-api-access-j629b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:57:48.493851 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.493821    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:48.501376 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.501355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:48.584597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.584565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume\") pod \"07373aed-2c05-476b-b5b7-4bf8f3325706\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") "
Jul 28 14:57:48.584785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.584723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume\") pod \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") "
Jul 28 14:57:48.584785 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.584758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbnsk\" (UniqueName: \"kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk\") pod \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\" (UID: \"2d7b78a7-3e08-45cb-8095-eb918955d8f9\") "
Jul 28 14:57:48.584871 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.584787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m4wm\" (UniqueName: \"kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm\") pod \"07373aed-2c05-476b-b5b7-4bf8f3325706\" (UID: \"07373aed-2c05-476b-b5b7-4bf8f3325706\") "
Jul 28 14:57:48.585065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.585048    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j629b\" (UniqueName: \"kubernetes.io/projected/b5985db3-a7a1-4442-aa31-427e46c8552e-kube-api-access-j629b\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.585112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.585079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07373aed-2c05-476b-b5b7-4bf8f3325706" (UID: "07373aed-2c05-476b-b5b7-4bf8f3325706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:57:48.585112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.585097    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5985db3-a7a1-4442-aa31-427e46c8552e-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.585112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.585112    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5985db3-a7a1-4442-aa31-427e46c8552e-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.585226 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.585132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d7b78a7-3e08-45cb-8095-eb918955d8f9" (UID: "2d7b78a7-3e08-45cb-8095-eb918955d8f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:57:48.586936 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.586905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk" (OuterVolumeSpecName: "kube-api-access-cbnsk") pod "2d7b78a7-3e08-45cb-8095-eb918955d8f9" (UID: "2d7b78a7-3e08-45cb-8095-eb918955d8f9"). InnerVolumeSpecName "kube-api-access-cbnsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:57:48.587009 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.586978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm" (OuterVolumeSpecName: "kube-api-access-5m4wm") pod "07373aed-2c05-476b-b5b7-4bf8f3325706" (UID: "07373aed-2c05-476b-b5b7-4bf8f3325706"). InnerVolumeSpecName "kube-api-access-5m4wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:57:48.686076 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.685998    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7b78a7-3e08-45cb-8095-eb918955d8f9-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.686076 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.686028    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnsk\" (UniqueName: \"kubernetes.io/projected/2d7b78a7-3e08-45cb-8095-eb918955d8f9-kube-api-access-cbnsk\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.686076 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.686038    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m4wm\" (UniqueName: \"kubernetes.io/projected/07373aed-2c05-476b-b5b7-4bf8f3325706-kube-api-access-5m4wm\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:48.686076 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:48.686047    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07373aed-2c05-476b-b5b7-4bf8f3325706-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:57:49.376162 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.376132    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"
Jul 28 14:57:49.376162 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.376137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp" event={"ID":"2d7b78a7-3e08-45cb-8095-eb918955d8f9","Type":"ContainerDied","Data":"60523c8bce0d8c421907b5e783874a3f2178c80b0c630432e5e44307975fddd6"}
Jul 28 14:57:49.376712 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.376204    2577 scope.go:117] "RemoveContainer" containerID="2cf7e49fe442ca93383dc5f829194e3e83130edc49e26bc20be1f8aae051db4b"
Jul 28 14:57:49.377480 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.377455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"
Jul 28 14:57:49.377605 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.377455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd" event={"ID":"b5985db3-a7a1-4442-aa31-427e46c8552e","Type":"ContainerDied","Data":"8e74d44cb8646e8b9173759d335e12d170990a54511df310db149ee328d25606"}
Jul 28 14:57:49.378947 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.378917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs" event={"ID":"07373aed-2c05-476b-b5b7-4bf8f3325706","Type":"ContainerDied","Data":"6280555ce8e6800e61e8fbb4c624ec7a00f7e037a969a1c98644b82377177d18"}
Jul 28 14:57:49.379059 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.378953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"
Jul 28 14:57:49.385511 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.385489    2577 scope.go:117] "RemoveContainer" containerID="6453174c4d42b77125665777d661209e1dffc85fd554f44bebd9a8e0c59c4e7d"
Jul 28 14:57:49.392832 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.392813    2577 scope.go:117] "RemoveContainer" containerID="dc66eb40a9e2f83eb6d074e64948d7dc6ce9e7c69d0d68f79c92943b5e299667"
Jul 28 14:57:49.396515 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.396493    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:49.403165 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.403144    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-efb491e3-84f86d747b-28qhs"]
Jul 28 14:57:49.412831 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.412786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:49.416310 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.416292    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-efb491e3-69c9b48d4b-7cqbp"]
Jul 28 14:57:49.425317 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.425294    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:49.428870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:49.428848    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-efb491e3-c9b559b49-6txtd"]
Jul 28 14:57:50.785794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:50.785761    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07373aed-2c05-476b-b5b7-4bf8f3325706" path="/var/lib/kubelet/pods/07373aed-2c05-476b-b5b7-4bf8f3325706/volumes"
Jul 28 14:57:50.786169 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:50.786129    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" path="/var/lib/kubelet/pods/2d7b78a7-3e08-45cb-8095-eb918955d8f9/volumes"
Jul 28 14:57:50.786469 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:50.786456    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5985db3-a7a1-4442-aa31-427e46c8552e" path="/var/lib/kubelet/pods/b5985db3-a7a1-4442-aa31-427e46c8552e/volumes"
Jul 28 14:57:54.701849 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:54.701814    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:57:54.702419 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:54.702092    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" containerID="cri-o://133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085" gracePeriod=30
Jul 28 14:57:55.520576 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:55.520522    2577 patch_prober.go:28] interesting pod/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 14:57:55.520741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:55.520605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 14:57:59.958442 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:57:59.958411    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:58:00.086858 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.086767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9ctg\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.086858 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.086814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.086858 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.086836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.086955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.086994    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087121 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket\") pod \"79fc71b4-73bd-45f5-9c4b-4f678b059496\" (UID: \"79fc71b4-73bd-45f5-9c4b-4f678b059496\") "
Jul 28 14:58:00.087373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:58:00.087373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:58:00.087373 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087238    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:58:00.087602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:58:00.087661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087630    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79fc71b4-73bd-45f5-9c4b-4f678b059496-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.087661 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087650    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.087766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087663    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.087766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087677    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.088020 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.087994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data" (OuterVolumeSpecName: "istio-data") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:58:00.089510 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.089482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 14:58:00.089591 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.089500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg" (OuterVolumeSpecName: "kube-api-access-x9ctg") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "kube-api-access-x9ctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:58:00.089637 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.089603    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:58:00.089692 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.089676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token" (OuterVolumeSpecName: "istio-token") pod "79fc71b4-73bd-45f5-9c4b-4f678b059496" (UID: "79fc71b4-73bd-45f5-9c4b-4f678b059496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:58:00.188259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.188226    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.188259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.188252    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.188259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.188261    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.188259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.188270    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9ctg\" (UniqueName: \"kubernetes.io/projected/79fc71b4-73bd-45f5-9c4b-4f678b059496-kube-api-access-x9ctg\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.188525 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.188279    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79fc71b4-73bd-45f5-9c4b-4f678b059496-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:58:00.434606 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.434510    2577 generic.go:356] "Generic (PLEG): container finished" podID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerID="133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085" exitCode=0
Jul 28 14:58:00.434606 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.434567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerDied","Data":"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"}
Jul 28 14:58:00.434606 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.434582    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"
Jul 28 14:58:00.434863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.434620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq" event={"ID":"79fc71b4-73bd-45f5-9c4b-4f678b059496","Type":"ContainerDied","Data":"22b74f6fb6da7a68d74848ac92b6f308e3a345a303c3d3f18a8d3b420ec9025b"}
Jul 28 14:58:00.434863 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.434644    2577 scope.go:117] "RemoveContainer" containerID="133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"
Jul 28 14:58:00.443482 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.443465    2577 scope.go:117] "RemoveContainer" containerID="54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"
Jul 28 14:58:00.451309 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.451289    2577 scope.go:117] "RemoveContainer" containerID="133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"
Jul 28 14:58:00.451595 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:58:00.451570    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085\": container with ID starting with 133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085 not found: ID does not exist" containerID="133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"
Jul 28 14:58:00.451709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.451603    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085"} err="failed to get container status \"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085\": rpc error: code = NotFound desc = could not find container \"133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085\": container with ID starting with 133a6af8b7c013642eaf30453254c0190a343e579e637a4fc24ce3f1dcf5b085 not found: ID does not exist"
Jul 28 14:58:00.451709 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.451627    2577 scope.go:117] "RemoveContainer" containerID="54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"
Jul 28 14:58:00.451881 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:58:00.451859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a\": container with ID starting with 54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a not found: ID does not exist" containerID="54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"
Jul 28 14:58:00.451920 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.451887    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a"} err="failed to get container status \"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a\": rpc error: code = NotFound desc = could not find container \"54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a\": container with ID starting with 54d401b4274ca8729ec9d224aba37505a29259cd620403f6ffdb5b8a8ec5ec2a not found: ID does not exist"
Jul 28 14:58:00.456687 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.456666    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:58:00.462500 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.462470    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-efb491e3-openshift-default-7bcb656ddd-9nkqq"]
Jul 28 14:58:00.785194 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:00.785160    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" path="/var/lib/kubelet/pods/79fc71b4-73bd-45f5-9c4b-4f678b059496/volumes"
Jul 28 14:58:15.087694 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.087653    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 14:58:15.088169 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088153    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5985db3-a7a1-4442-aa31-427e46c8552e" containerName="maas-api"
Jul 28 14:58:15.088211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088172    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5985db3-a7a1-4442-aa31-427e46c8552e" containerName="maas-api"
Jul 28 14:58:15.088211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088186    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:15.088211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088196    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:15.088305 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088226    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07373aed-2c05-476b-b5b7-4bf8f3325706" containerName="payload-pre-processing"
Jul 28 14:58:15.088305 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="07373aed-2c05-476b-b5b7-4bf8f3325706" containerName="payload-pre-processing"
Jul 28 14:58:15.088305 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088249    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" containerName="payload-processing"
Jul 28 14:58:15.088305 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088258    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" containerName="payload-processing"
Jul 28 14:58:15.088481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088338    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="07373aed-2c05-476b-b5b7-4bf8f3325706" containerName="payload-pre-processing"
Jul 28 14:58:15.088481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088350    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7b78a7-3e08-45cb-8095-eb918955d8f9" containerName="payload-processing"
Jul 28 14:58:15.088481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088363    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5985db3-a7a1-4442-aa31-427e46c8552e" containerName="maas-api"
Jul 28 14:58:15.088481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088374    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:15.088481 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.088384    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:15.093345 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.093322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.095673 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.095648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 14:58:15.095814 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.095693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-96a65772-openshift-default-dockercfg-k8fqv\""
Jul 28 14:58:15.103602 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.103578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 14:58:15.222584 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222584 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjgq\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222794 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.222994 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.222802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.323637 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.323842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.323842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.323842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.323842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.323978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324061 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjgq\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324221 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324293 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.324512 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.324433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.326256 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.326234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.326358 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.326327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.331765 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.331743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.332066 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.332045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjgq\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq\") pod \"e2e-mt-96a65772-openshift-default-546fc9768-l9t7w\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") " pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.404757 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.404675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:15.540347 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.540320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 14:58:15.542985 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:58:15.542955    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5037ab_998f_42f6_8ba7_6481e7dca4eb.slice/crio-27b1187a6126e23b198096784bb38d0345be20d77909ab9ea02c848c89fc9375 WatchSource:0}: Error finding container 27b1187a6126e23b198096784bb38d0345be20d77909ab9ea02c848c89fc9375: Status 404 returned error can't find the container with id 27b1187a6126e23b198096784bb38d0345be20d77909ab9ea02c848c89fc9375
Jul 28 14:58:15.545190 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.545154    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:15.545270 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.545251    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:15.545311 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:15.545300    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:16.509518 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:16.509476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerStarted","Data":"08aba441fa58b4af794d65a2fa7f87878176e7e533ae1e4d8700c64eac82e5a1"}
Jul 28 14:58:16.510027 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:16.509579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerStarted","Data":"27b1187a6126e23b198096784bb38d0345be20d77909ab9ea02c848c89fc9375"}
Jul 28 14:58:16.528913 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:16.528859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podStartSLOduration=1.528843288 podStartE2EDuration="1.528843288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:16.527075641 +0000 UTC m=+2134.213153756" watchObservedRunningTime="2026-07-28 14:58:16.528843288 +0000 UTC m=+2134.214921400"
Jul 28 14:58:17.405283 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:17.405235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:17.410384 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:17.410357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:17.514548 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:17.514518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:17.515799 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:17.515778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:18.565077 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.565045    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:58:18.565617 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.565600    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:18.565677 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.565620    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fc71b4-73bd-45f5-9c4b-4f678b059496" containerName="istio-proxy"
Jul 28 14:58:18.569035 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.569019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.571485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.571464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 14:58:18.571630 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.571470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-96a65772-serving-cert\""
Jul 28 14:58:18.580922 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.580894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:58:18.654842 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.654807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxsn\" (UniqueName: \"kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.655048 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.654872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.655048 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.655034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.756396 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.756355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.756679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.756441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxsn\" (UniqueName: \"kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.756679 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.756487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.756679 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:58:18.756518    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-serving-cert: secret "maas-api-e2e-mt-96a65772-serving-cert" not found
Jul 28 14:58:18.756679 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:58:18.756619    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls podName:8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c nodeName:}" failed. No retries permitted until 2026-07-28 14:58:19.256601444 +0000 UTC m=+2136.942679535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls") pod "maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" (UID: "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c") : secret "maas-api-e2e-mt-96a65772-serving-cert" not found
Jul 28 14:58:18.757201 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.757177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:18.768944 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:18.768911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxsn\" (UniqueName: \"kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:19.261133 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.261098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:19.263926 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.263892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") pod \"maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:19.344210 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.344172    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:58:19.347819 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.347798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.350058 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.350030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-96a65772\""
Jul 28 14:58:19.357349 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.357327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:58:19.382467 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.382436    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:58:19.385727 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.385707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.388065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.388043    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-96a65772-dockercfg-g9hpp\""
Jul 28 14:58:19.396437 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.396417    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:58:19.462899 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.462866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.463112 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.462930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g454p\" (UniqueName: \"kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.480821 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.480789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:19.564278 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.564241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.564465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.564292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmkf\" (UniqueName: \"kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.564465 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.564321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g454p\" (UniqueName: \"kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.564597 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.564497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.565147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.565129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.573170 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.573136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g454p\" (UniqueName: \"kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p\") pod \"payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.621364 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.621319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:58:19.623880 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:58:19.623853    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b80bbc5_1f7e_4240_98b8_fb99c76a7b3c.slice/crio-edc9a0aad50577944607edb9b875bbb41ef89bcd6ee083041408808fcbc97e63 WatchSource:0}: Error finding container edc9a0aad50577944607edb9b875bbb41ef89bcd6ee083041408808fcbc97e63: Status 404 returned error can't find the container with id edc9a0aad50577944607edb9b875bbb41ef89bcd6ee083041408808fcbc97e63
Jul 28 14:58:19.625805 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.625786    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:58:19.659276 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.659244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:19.665685 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.665655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.665807 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.665789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmkf\" (UniqueName: \"kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.666495 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.666475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.674036 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.674014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmkf\" (UniqueName: \"kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf\") pod \"payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") " pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.696984 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.696955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:19.803811 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.803764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:58:19.806161 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:58:19.806131    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb540f999_4baf_4eed_8cc8_bb59dbf03dd9.slice/crio-5eff42ebb7e787c04e1545c4c39d07aa2731599cc04f57b6875f8e12497a2965 WatchSource:0}: Error finding container 5eff42ebb7e787c04e1545c4c39d07aa2731599cc04f57b6875f8e12497a2965: Status 404 returned error can't find the container with id 5eff42ebb7e787c04e1545c4c39d07aa2731599cc04f57b6875f8e12497a2965
Jul 28 14:58:19.859622 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:19.859595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:58:19.867734 ip-10-0-142-230 kubenswrapper[2577]: W0728 14:58:19.867705    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69846d3_b40d_4267_afee_ef5cd470040b.slice/crio-9c0b7ccba16eaee925a14d182ae506d9f3a43800ac95ee05ce60c979b4c420f0 WatchSource:0}: Error finding container 9c0b7ccba16eaee925a14d182ae506d9f3a43800ac95ee05ce60c979b4c420f0: Status 404 returned error can't find the container with id 9c0b7ccba16eaee925a14d182ae506d9f3a43800ac95ee05ce60c979b4c420f0
Jul 28 14:58:20.532182 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.532146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" event={"ID":"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c","Type":"ContainerStarted","Data":"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"}
Jul 28 14:58:20.532424 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.532236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" event={"ID":"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c","Type":"ContainerStarted","Data":"edc9a0aad50577944607edb9b875bbb41ef89bcd6ee083041408808fcbc97e63"}
Jul 28 14:58:20.532424 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.532267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:20.533932 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.533903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" event={"ID":"b69846d3-b40d-4267-afee-ef5cd470040b","Type":"ContainerStarted","Data":"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"}
Jul 28 14:58:20.534065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.533952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" event={"ID":"b69846d3-b40d-4267-afee-ef5cd470040b","Type":"ContainerStarted","Data":"9c0b7ccba16eaee925a14d182ae506d9f3a43800ac95ee05ce60c979b4c420f0"}
Jul 28 14:58:20.534065 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.533988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:20.535591 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.535562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" event={"ID":"b540f999-4baf-4eed-8cc8-bb59dbf03dd9","Type":"ContainerStarted","Data":"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"}
Jul 28 14:58:20.535741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.535610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" event={"ID":"b540f999-4baf-4eed-8cc8-bb59dbf03dd9","Type":"ContainerStarted","Data":"5eff42ebb7e787c04e1545c4c39d07aa2731599cc04f57b6875f8e12497a2965"}
Jul 28 14:58:20.535741 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.535663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:20.548056 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.548005    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" podStartSLOduration=2.072759136 podStartE2EDuration="2.547987799s" totalImagesPullingTime="475.228663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:20.547016144 +0000 UTC m=+2138.233094256" watchObservedRunningTime="2026-07-28 14:58:20.547987799 +0000 UTC m=+2138.234065914"
Jul 28 14:58:20.563377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.563327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" podStartSLOduration=1.563314101 podStartE2EDuration="1.563314101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:20.561577341 +0000 UTC m=+2138.247655455" watchObservedRunningTime="2026-07-28 14:58:20.563314101 +0000 UTC m=+2138.249392213"
Jul 28 14:58:20.575758 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:20.575704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" podStartSLOduration=1.5756890399999999 podStartE2EDuration="1.57568904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:20.57521042 +0000 UTC m=+2138.261288534" watchObservedRunningTime="2026-07-28 14:58:20.57568904 +0000 UTC m=+2138.261767179"
Jul 28 14:58:26.546174 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:26.546149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:58:31.542738 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:31.542708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:58:31.543211 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:31.543028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:58:32.572676 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.572646    2577 patch_prober.go:28] interesting pod/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:52504->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Jul 28 14:58:32.573101 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.572707    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:52504->10.133.0.58:15021: read: connection reset by peer"
Jul 28 14:58:32.594870 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.594848    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerID="08aba441fa58b4af794d65a2fa7f87878176e7e533ae1e4d8700c64eac82e5a1" exitCode=0
Jul 28 14:58:32.594970 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.594879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerDied","Data":"08aba441fa58b4af794d65a2fa7f87878176e7e533ae1e4d8700c64eac82e5a1"}
Jul 28 14:58:32.595252 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.595239    2577 scope.go:117] "RemoveContainer" containerID="08aba441fa58b4af794d65a2fa7f87878176e7e533ae1e4d8700c64eac82e5a1"
Jul 28 14:58:32.596201 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.596173    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:32.596275 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.596243    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:32.596314 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:32.596275    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:58:33.601164 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:33.601123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerStarted","Data":"a048466c5f1208a31a5b82cbc9e494e89229327c7e9ff39bb1f1b802ca222861"}
Jul 28 14:58:34.405652 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:34.405570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:34.405811 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:34.405774    2577 patch_prober.go:28] interesting pod/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 28 14:58:34.405856 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:34.405831    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 28 14:58:35.405747 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:35.405712    2577 patch_prober.go:28] interesting pod/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 28 14:58:35.406141 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:35.405769    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 28 14:58:36.405618 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:36.405579    2577 patch_prober.go:28] interesting pod/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 28 14:58:36.405802 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:36.405639    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 28 14:58:37.405226 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:37.405189    2577 patch_prober.go:28] interesting pod/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 28 14:58:37.405407 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:37.405246    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 28 14:58:38.409392 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:38.409361    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:38.409851 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:38.409630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:58:38.410391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:58:38.410373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 14:59:28.869255 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.869209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:59:28.869828 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.869573    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" podUID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" containerName="maas-api" containerID="cri-o://2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169" gracePeriod=30
Jul 28 14:59:28.925200 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.925165    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:59:28.925974 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.925400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" podUID="b69846d3-b40d-4267-afee-ef5cd470040b" containerName="payload-processing" containerID="cri-o://b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755" gracePeriod=30
Jul 28 14:59:28.934582 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.934548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:59:28.934820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:28.934781    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" podUID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" containerName="payload-pre-processing" containerID="cri-o://1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2" gracePeriod=30
Jul 28 14:59:29.198625 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.198586    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:59:29.233485 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.233460    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:59:29.236927 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.236898    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:59:29.325766 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca\") pod \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") "
Jul 28 14:59:29.325968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume\") pod \"b69846d3-b40d-4267-afee-ef5cd470040b\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") "
Jul 28 14:59:29.325968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume\") pod \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") "
Jul 28 14:59:29.325968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcmkf\" (UniqueName: \"kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf\") pod \"b69846d3-b40d-4267-afee-ef5cd470040b\" (UID: \"b69846d3-b40d-4267-afee-ef5cd470040b\") "
Jul 28 14:59:29.325968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") pod \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") "
Jul 28 14:59:29.325968 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g454p\" (UniqueName: \"kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p\") pod \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\" (UID: \"b540f999-4baf-4eed-8cc8-bb59dbf03dd9\") "
Jul 28 14:59:29.326259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.325975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqxsn\" (UniqueName: \"kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn\") pod \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\" (UID: \"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c\") "
Jul 28 14:59:29.326259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.326153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" (UID: "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:59:29.326259 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.326246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b540f999-4baf-4eed-8cc8-bb59dbf03dd9" (UID: "b540f999-4baf-4eed-8cc8-bb59dbf03dd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:59:29.326503 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.326476    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.326647 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.326580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b69846d3-b40d-4267-afee-ef5cd470040b" (UID: "b69846d3-b40d-4267-afee-ef5cd470040b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:59:29.328310 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.328277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" (UID: "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:59:29.328437 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.328309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn" (OuterVolumeSpecName: "kube-api-access-nqxsn") pod "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" (UID: "8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c"). InnerVolumeSpecName "kube-api-access-nqxsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:59:29.328513 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.328494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p" (OuterVolumeSpecName: "kube-api-access-g454p") pod "b540f999-4baf-4eed-8cc8-bb59dbf03dd9" (UID: "b540f999-4baf-4eed-8cc8-bb59dbf03dd9"). InnerVolumeSpecName "kube-api-access-g454p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:59:29.328582 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.328569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf" (OuterVolumeSpecName: "kube-api-access-wcmkf") pod "b69846d3-b40d-4267-afee-ef5cd470040b" (UID: "b69846d3-b40d-4267-afee-ef5cd470040b"). InnerVolumeSpecName "kube-api-access-wcmkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427284    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcmkf\" (UniqueName: \"kubernetes.io/projected/b69846d3-b40d-4267-afee-ef5cd470040b-kube-api-access-wcmkf\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427318    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427330    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g454p\" (UniqueName: \"kubernetes.io/projected/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-kube-api-access-g454p\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427340    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxsn\" (UniqueName: \"kubernetes.io/projected/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c-kube-api-access-nqxsn\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427349    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69846d3-b40d-4267-afee-ef5cd470040b-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.427377 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.427358    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b540f999-4baf-4eed-8cc8-bb59dbf03dd9-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 14:59:29.873212 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.873174    2577 generic.go:356] "Generic (PLEG): container finished" podID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" containerID="2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169" exitCode=0
Jul 28 14:59:29.873708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.873226    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"
Jul 28 14:59:29.873708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.873257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" event={"ID":"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c","Type":"ContainerDied","Data":"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"}
Jul 28 14:59:29.873708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.873316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl" event={"ID":"8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c","Type":"ContainerDied","Data":"edc9a0aad50577944607edb9b875bbb41ef89bcd6ee083041408808fcbc97e63"}
Jul 28 14:59:29.873708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.873347    2577 scope.go:117] "RemoveContainer" containerID="2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"
Jul 28 14:59:29.874790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.874634    2577 generic.go:356] "Generic (PLEG): container finished" podID="b69846d3-b40d-4267-afee-ef5cd470040b" containerID="b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755" exitCode=0
Jul 28 14:59:29.874790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.874708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"
Jul 28 14:59:29.874790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.874715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" event={"ID":"b69846d3-b40d-4267-afee-ef5cd470040b","Type":"ContainerDied","Data":"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"}
Jul 28 14:59:29.874790 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.874767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6" event={"ID":"b69846d3-b40d-4267-afee-ef5cd470040b","Type":"ContainerDied","Data":"9c0b7ccba16eaee925a14d182ae506d9f3a43800ac95ee05ce60c979b4c420f0"}
Jul 28 14:59:29.876864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.876806    2577 generic.go:356] "Generic (PLEG): container finished" podID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" containerID="1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2" exitCode=0
Jul 28 14:59:29.876864 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.876839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" event={"ID":"b540f999-4baf-4eed-8cc8-bb59dbf03dd9","Type":"ContainerDied","Data":"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"}
Jul 28 14:59:29.877024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.876863    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"
Jul 28 14:59:29.877024 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.876882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn" event={"ID":"b540f999-4baf-4eed-8cc8-bb59dbf03dd9","Type":"ContainerDied","Data":"5eff42ebb7e787c04e1545c4c39d07aa2731599cc04f57b6875f8e12497a2965"}
Jul 28 14:59:29.884762 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.884725    2577 scope.go:117] "RemoveContainer" containerID="2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"
Jul 28 14:59:29.889098 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:59:29.889067    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169\": container with ID starting with 2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169 not found: ID does not exist" containerID="2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"
Jul 28 14:59:29.889286 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.889263    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169"} err="failed to get container status \"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169\": rpc error: code = NotFound desc = could not find container \"2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169\": container with ID starting with 2a73f7ebb74a212a3aed2dd959f2f6417b3002daa1633a054c70e82770e4e169 not found: ID does not exist"
Jul 28 14:59:29.889391 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.889380    2577 scope.go:117] "RemoveContainer" containerID="b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"
Jul 28 14:59:29.900317 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.900292    2577 scope.go:117] "RemoveContainer" containerID="b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"
Jul 28 14:59:29.900659 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:59:29.900628    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755\": container with ID starting with b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755 not found: ID does not exist" containerID="b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"
Jul 28 14:59:29.900764 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.900670    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755"} err="failed to get container status \"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755\": rpc error: code = NotFound desc = could not find container \"b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755\": container with ID starting with b5b34584a58dd2be7b504c48025a6c338bfd4dbc121759979efe231583aa2755 not found: ID does not exist"
Jul 28 14:59:29.900764 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.900689    2577 scope.go:117] "RemoveContainer" containerID="1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"
Jul 28 14:59:29.907147 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.907119    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:59:29.909803 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.909784    2577 scope.go:117] "RemoveContainer" containerID="1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"
Jul 28 14:59:29.910151 ip-10-0-142-230 kubenswrapper[2577]: E0728 14:59:29.910112    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2\": container with ID starting with 1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2 not found: ID does not exist" containerID="1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"
Jul 28 14:59:29.910204 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.910160    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2"} err="failed to get container status \"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2\": rpc error: code = NotFound desc = could not find container \"1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2\": container with ID starting with 1ac70fbd59347a1b1fcf1140d3cdee199e0cd5a68c5aeb7257c9099b83a6ddc2 not found: ID does not exist"
Jul 28 14:59:29.910967 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.910946    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-96a65772-77b8cc8b4c-lz8vl"]
Jul 28 14:59:29.919820 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.919786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:59:29.922965 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.922937    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-96a65772-7dd4bc9476-hdtvn"]
Jul 28 14:59:29.931469 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.931435    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:59:29.935169 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:29.935145    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-96a65772-d7964bd66-mxzm6"]
Jul 28 14:59:30.786041 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:30.786010    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" path="/var/lib/kubelet/pods/8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c/volumes"
Jul 28 14:59:30.786368 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:30.786356    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" path="/var/lib/kubelet/pods/b540f999-4baf-4eed-8cc8-bb59dbf03dd9/volumes"
Jul 28 14:59:30.786708 ip-10-0-142-230 kubenswrapper[2577]: I0728 14:59:30.786695    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69846d3-b40d-4267-afee-ef5cd470040b" path="/var/lib/kubelet/pods/b69846d3-b40d-4267-afee-ef5cd470040b/volumes"
Jul 28 15:00:00.136374 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"]
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136749    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" containerName="payload-pre-processing"
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136765    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" containerName="payload-pre-processing"
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136775    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" containerName="maas-api"
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136781    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" containerName="maas-api"
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136803    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69846d3-b40d-4267-afee-ef5cd470040b" containerName="payload-processing"
Jul 28 15:00:00.136839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136809    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69846d3-b40d-4267-afee-ef5cd470040b" containerName="payload-processing"
Jul 28 15:00:00.137033 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b80bbc5-1f7e-4240-98b8-fb99c76a7b3c" containerName="maas-api"
Jul 28 15:00:00.137033 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136880    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69846d3-b40d-4267-afee-ef5cd470040b" containerName="payload-processing"
Jul 28 15:00:00.137033 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.136889    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b540f999-4baf-4eed-8cc8-bb59dbf03dd9" containerName="payload-pre-processing"
Jul 28 15:00:00.140133 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.140114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"
Jul 28 15:00:00.142474 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.142456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:00:00.145739 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.145715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"]
Jul 28 15:00:00.152911 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.152889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"
Jul 28 15:00:00.280651 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:00.280619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"]
Jul 28 15:00:00.283562 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:00:00.283509    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14313d4_8aac_4fc2_b16d_e2b7c30cf6cc.slice/crio-725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3 WatchSource:0}: Error finding container 725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3: Status 404 returned error can't find the container with id 725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3
Jul 28 15:00:01.035766 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:01.035735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s" event={"ID":"e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc","Type":"ContainerStarted","Data":"725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3"}
Jul 28 15:00:04.050928 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:04.050834    2577 generic.go:356] "Generic (PLEG): container finished" podID="e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc" containerID="e58cc7f218f0c7b79cd972876f93e21da7d7c883babfb2029b84f965bf4387b6" exitCode=0
Jul 28 15:00:04.050928 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:04.050897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s" event={"ID":"e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc","Type":"ContainerDied","Data":"e58cc7f218f0c7b79cd972876f93e21da7d7c883babfb2029b84f965bf4387b6"}
Jul 28 15:00:05.186416 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:05.186391    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"
Jul 28 15:00:06.060468 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:06.060438    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s"
Jul 28 15:00:06.060468 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:06.060447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-nqz7s" event={"ID":"e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc","Type":"ContainerDied","Data":"725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3"}
Jul 28 15:00:06.060707 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:06.060497    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725cb604bac55cabf2c929f330c1b91c77d75a4f52da9350044f60c5a47481b3"
Jul 28 15:00:44.085801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:44.085758    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 15:00:44.086446 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:44.086036    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy" containerID="cri-o://a048466c5f1208a31a5b82cbc9e494e89229327c7e9ff39bb1f1b802ca222861" gracePeriod=30
Jul 28 15:00:49.264100 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.264067    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerID="a048466c5f1208a31a5b82cbc9e494e89229327c7e9ff39bb1f1b802ca222861" exitCode=0
Jul 28 15:00:49.264508 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.264169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerDied","Data":"a048466c5f1208a31a5b82cbc9e494e89229327c7e9ff39bb1f1b802ca222861"}
Jul 28 15:00:49.264508 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.264251    2577 scope.go:117] "RemoveContainer" containerID="08aba441fa58b4af794d65a2fa7f87878176e7e533ae1e4d8700c64eac82e5a1"
Jul 28 15:00:49.351150 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.351088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 15:00:49.466289 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466470 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466470 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466469    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:00:49.466588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hjgq\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466781 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:00:49.466781 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:00:49.466886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.466886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.466840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo\") pod \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\" (UID: \"4f5037ab-998f-42f6-8ba7-6481e7dca4eb\") "
Jul 28 15:00:49.467142 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.467114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:00:49.467250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.467184    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.467250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.467203    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.467250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.467218    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.467467 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.467442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:00:49.468831 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.468805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq" (OuterVolumeSpecName: "kube-api-access-2hjgq") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "kube-api-access-2hjgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:00:49.468946 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.468839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:00:49.469081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.469059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:00:49.469081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.469069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f5037ab-998f-42f6-8ba7-6481e7dca4eb" (UID: "4f5037ab-998f-42f6-8ba7-6481e7dca4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:00:49.567895 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567858    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.567895 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567890    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.568181 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567913    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjgq\" (UniqueName: \"kubernetes.io/projected/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-kube-api-access-2hjgq\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.568181 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567929    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.568181 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567941    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:49.568181 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:49.567979    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5037ab-998f-42f6-8ba7-6481e7dca4eb-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:00:50.270735 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.270695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w" event={"ID":"4f5037ab-998f-42f6-8ba7-6481e7dca4eb","Type":"ContainerDied","Data":"27b1187a6126e23b198096784bb38d0345be20d77909ab9ea02c848c89fc9375"}
Jul 28 15:00:50.270735 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.270716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"
Jul 28 15:00:50.271248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.270766    2577 scope.go:117] "RemoveContainer" containerID="a048466c5f1208a31a5b82cbc9e494e89229327c7e9ff39bb1f1b802ca222861"
Jul 28 15:00:50.294370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.294332    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 15:00:50.304927 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.304896    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-96a65772-openshift-default-546fc9768-l9t7w"]
Jul 28 15:00:50.785336 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:00:50.785299    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" path="/var/lib/kubelet/pods/4f5037ab-998f-42f6-8ba7-6481e7dca4eb/volumes"
Jul 28 15:01:43.564736 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.564704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:01:43.565217 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:43.565217 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565178    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:43.565305 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565220    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc" containerName="cleanup"
Jul 28 15:01:43.565305 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc" containerName="cleanup"
Jul 28 15:01:43.565370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565312    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:43.565370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565325    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc" containerName="cleanup"
Jul 28 15:01:43.565370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.565333    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:43.569424 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.569406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.571737 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.571716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d392e9-openshift-default-dockercfg-4gckq\""
Jul 28 15:01:43.571737 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.571722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 15:01:43.579520 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.579493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:01:43.674105 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674315 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674315 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674315 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674465 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674465 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674465 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674465 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2d6\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.674647 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.674486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775762 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.775967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.775917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776340 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776340 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2d6\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776340 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776340 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776591 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776591 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776591 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.776780 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.776761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.778316 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.778287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.778427 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.778377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.782869 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.782844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.782974 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.782957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2d6\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6\") pod \"e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") " pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:43.882216 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:43.882113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:44.022835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.022808    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:01:44.024085 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:01:44.024049    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940fbf67_d361_47b7_b8a1_ef84aa6387d6.slice/crio-10a6179f02e7b1d9a2750109530dbeab932930057f56f43089ba0bf0f3df1377 WatchSource:0}: Error finding container 10a6179f02e7b1d9a2750109530dbeab932930057f56f43089ba0bf0f3df1377: Status 404 returned error can't find the container with id 10a6179f02e7b1d9a2750109530dbeab932930057f56f43089ba0bf0f3df1377
Jul 28 15:01:44.026414 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.026343    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:01:44.026494 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.026416    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:01:44.026494 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.026476    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:01:44.516371 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.516329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerStarted","Data":"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"}
Jul 28 15:01:44.516615 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.516414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerStarted","Data":"10a6179f02e7b1d9a2750109530dbeab932930057f56f43089ba0bf0f3df1377"}
Jul 28 15:01:44.533907 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.533846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podStartSLOduration=1.53382463 podStartE2EDuration="1.53382463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:01:44.533404167 +0000 UTC m=+2342.219482281" watchObservedRunningTime="2026-07-28 15:01:44.53382463 +0000 UTC m=+2342.219902741"
Jul 28 15:01:44.882735 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:44.882639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:45.887415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:45.887378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:46.526691 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.526653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:46.527650 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.527627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:01:46.776792 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.776701    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:01:46.777266 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.777251    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:46.777308 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.777270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5037ab-998f-42f6-8ba7-6481e7dca4eb" containerName="istio-proxy"
Jul 28 15:01:46.780924 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.780895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:46.783061 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.783037    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d392e9-serving-cert\""
Jul 28 15:01:46.783197 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.783104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 15:01:46.789489 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.789460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:01:46.904597 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.904561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:46.904988 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.904648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:46.904988 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:46.904684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9p7\" (UniqueName: \"kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.005957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.005924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.006136 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.005972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9p7\" (UniqueName: \"kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.006136 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.006015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.006208 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:01:47.006142    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-serving-cert: secret "maas-api-e2e-inf-a-d392e9-serving-cert" not found
Jul 28 15:01:47.006243 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:01:47.006209    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls podName:3636bab2-2310-4f8e-a03b-a9b67e3091ec nodeName:}" failed. No retries permitted until 2026-07-28 15:01:47.506190069 +0000 UTC m=+2345.192268176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls") pod "maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" (UID: "3636bab2-2310-4f8e-a03b-a9b67e3091ec") : secret "maas-api-e2e-inf-a-d392e9-serving-cert" not found
Jul 28 15:01:47.006622 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.006603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.014393 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.014364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9p7\" (UniqueName: \"kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.355003 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.354967    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:01:47.358643 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.358621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.360767 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.360744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d392e9\""
Jul 28 15:01:47.370656 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.370630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:01:47.394992 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.394962    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:01:47.398329 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.398297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.403324 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.403303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d392e9-dockercfg-hn8n2\""
Jul 28 15:01:47.413953 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.413921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:01:47.509984 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.509945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.510194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.510016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.510194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.510069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbvv\" (UniqueName: \"kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.510194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.510125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.510334 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.510202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.512820 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.512793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.611764 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.611673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.611764 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.611725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbvv\" (UniqueName: \"kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.611994 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.611836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.611994 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.611881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.612527 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.612507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.612651 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.612626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.619260 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.619237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbvv\" (UniqueName: \"kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv\") pod \"payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.619390 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.619275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm\") pod \"payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.670442 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.670408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:47.695392 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.695364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:47.709730 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.709693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:47.826496 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.826451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:01:47.828511 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:01:47.828476    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24508d7a_c673_44d7_83ad_6e541b768907.slice/crio-28e8af7bdc104178b9814a9e5e2ea6f26cbc9b17ec1d3c88604e5716176ff099 WatchSource:0}: Error finding container 28e8af7bdc104178b9814a9e5e2ea6f26cbc9b17ec1d3c88604e5716176ff099: Status 404 returned error can't find the container with id 28e8af7bdc104178b9814a9e5e2ea6f26cbc9b17ec1d3c88604e5716176ff099
Jul 28 15:01:47.859602 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.859570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:01:47.866248 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:01:47.866223    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3636bab2_2310_4f8e_a03b_a9b67e3091ec.slice/crio-002158d0f1efac3275141302867054902686d802b25680ec7d445bed030c7ca0 WatchSource:0}: Error finding container 002158d0f1efac3275141302867054902686d802b25680ec7d445bed030c7ca0: Status 404 returned error can't find the container with id 002158d0f1efac3275141302867054902686d802b25680ec7d445bed030c7ca0
Jul 28 15:01:47.890830 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:47.890803    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:01:47.892203 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:01:47.892182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcb9e13_9443_4beb_8338_fe39daaf2230.slice/crio-eae3ef556595c582d0df1b86e6c6a50cf5e28b3ef01581d0c7aaf4ff0e864b5f WatchSource:0}: Error finding container eae3ef556595c582d0df1b86e6c6a50cf5e28b3ef01581d0c7aaf4ff0e864b5f: Status 404 returned error can't find the container with id eae3ef556595c582d0df1b86e6c6a50cf5e28b3ef01581d0c7aaf4ff0e864b5f
Jul 28 15:01:48.539048 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.539010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" event={"ID":"24508d7a-c673-44d7-83ad-6e541b768907","Type":"ContainerStarted","Data":"36778335ed71e187f8c915d9e77824beb8c1cf9030abb722994f581bed398000"}
Jul 28 15:01:48.539667 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.539103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" event={"ID":"24508d7a-c673-44d7-83ad-6e541b768907","Type":"ContainerStarted","Data":"28e8af7bdc104178b9814a9e5e2ea6f26cbc9b17ec1d3c88604e5716176ff099"}
Jul 28 15:01:48.539667 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.539135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:01:48.540338 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.540314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" event={"ID":"3636bab2-2310-4f8e-a03b-a9b67e3091ec","Type":"ContainerStarted","Data":"002158d0f1efac3275141302867054902686d802b25680ec7d445bed030c7ca0"}
Jul 28 15:01:48.541738 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.541716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" event={"ID":"dbcb9e13-9443-4beb-8338-fe39daaf2230","Type":"ContainerStarted","Data":"f42c696c29c990c5eb0519a411d82093ab202b97540255a01d8cab9158625f1a"}
Jul 28 15:01:48.541856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.541757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" event={"ID":"dbcb9e13-9443-4beb-8338-fe39daaf2230","Type":"ContainerStarted","Data":"eae3ef556595c582d0df1b86e6c6a50cf5e28b3ef01581d0c7aaf4ff0e864b5f"}
Jul 28 15:01:48.553983 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.553937    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" podStartSLOduration=1.55392377 podStartE2EDuration="1.55392377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:01:48.552051826 +0000 UTC m=+2346.238129941" watchObservedRunningTime="2026-07-28 15:01:48.55392377 +0000 UTC m=+2346.240001860"
Jul 28 15:01:48.567192 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:48.567139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" podStartSLOduration=1.567122532 podStartE2EDuration="1.567122532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:01:48.565305911 +0000 UTC m=+2346.251384025" watchObservedRunningTime="2026-07-28 15:01:48.567122532 +0000 UTC m=+2346.253200645"
Jul 28 15:01:49.548032 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:49.547983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" event={"ID":"3636bab2-2310-4f8e-a03b-a9b67e3091ec","Type":"ContainerStarted","Data":"4753472b77f1101ce0e50aacf219e12480c4fc8dc2c386b5d5d682a37656ec1c"}
Jul 28 15:01:49.548449 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:49.548334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:49.548449 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:49.548372    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:01:49.564552 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:49.564492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" podStartSLOduration=2.454649446 podStartE2EDuration="3.564473484s" totalImagesPullingTime="1.109824038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:01:49.562440464 +0000 UTC m=+2347.248518579" watchObservedRunningTime="2026-07-28 15:01:49.564473484 +0000 UTC m=+2347.250551596"
Jul 28 15:01:55.557688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:55.557660    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:01:59.550824 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:01:59.550789    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:02:00.555143 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:00.555113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:02:01.539379 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.539345    2577 patch_prober.go:28] interesting pod/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:52800->10.133.0.63:15021: read: connection reset by peer" start-of-body=
Jul 28 15:02:01.539577 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.539426    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:52800->10.133.0.63:15021: read: connection reset by peer"
Jul 28 15:02:01.608467 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.608437    2577 generic.go:356] "Generic (PLEG): container finished" podID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerID="599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67" exitCode=0
Jul 28 15:02:01.608889 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.608512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerDied","Data":"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"}
Jul 28 15:02:01.608958 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.608944    2577 scope.go:117] "RemoveContainer" containerID="599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"
Jul 28 15:02:01.609986 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.609955    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:01.610046 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.610036    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:01.610093 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:01.610080    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:02.617837 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:02.617794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerStarted","Data":"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"}
Jul 28 15:02:02.883195 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:02.883096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:02:02.883353 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:02.883332    2577 patch_prober.go:28] interesting pod/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:02:02.883402 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:02.883381    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 28 15:02:03.883098 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:03.883062    2577 patch_prober.go:28] interesting pod/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:02:03.883506 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:03.883135    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 28 15:02:04.883173 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:04.883133    2577 patch_prober.go:28] interesting pod/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:02:04.883620 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:04.883196    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 28 15:02:05.883007 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:05.882971    2577 patch_prober.go:28] interesting pod/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:02:05.883189 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:05.883023    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 28 15:02:06.886476 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:06.886442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:02:06.886941 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:06.886862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:02:06.887526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:06.887507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:02:12.300641 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.300603    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:02:12.303412 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.303393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.305685 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.305664    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-904baa-openshift-default-dockercfg-pjlt8\""
Jul 28 15:02:12.315322 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.315298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:02:12.445354 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445354 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445354 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xm4\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.445815 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.445684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546594 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xm4\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.546801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547073 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547073 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547073 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.546866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547240 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.547094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547240 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.547161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547240 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.547203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547397 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.547300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.547462 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.547438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.549264 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.549238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.549494 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.549474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.554053 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.554029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.554326 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.554304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xm4\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4\") pod \"e2e-inf-b-904baa-openshift-default-59885b7858-c44r7\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") " pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.615528 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.615493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:12.752200 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.752174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:02:12.754235 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:02:12.754212    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62269da_cd45_481d_8d2c_98e27d238fb8.slice/crio-184204165f30ba58041b1c4b85f49f063c93747c63b7d6f473c069d68873460c WatchSource:0}: Error finding container 184204165f30ba58041b1c4b85f49f063c93747c63b7d6f473c069d68873460c: Status 404 returned error can't find the container with id 184204165f30ba58041b1c4b85f49f063c93747c63b7d6f473c069d68873460c
Jul 28 15:02:12.756314 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.756286    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:12.756379 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.756350    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:12.756415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:12.756376    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:13.677202 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:13.677168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerStarted","Data":"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"}
Jul 28 15:02:13.677858 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:13.677224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerStarted","Data":"184204165f30ba58041b1c4b85f49f063c93747c63b7d6f473c069d68873460c"}
Jul 28 15:02:13.695358 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:13.695304    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podStartSLOduration=1.695286684 podStartE2EDuration="1.695286684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:02:13.694305111 +0000 UTC m=+2371.380383251" watchObservedRunningTime="2026-07-28 15:02:13.695286684 +0000 UTC m=+2371.381364798"
Jul 28 15:02:14.616404 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:14.616365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:14.621859 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:14.621833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:14.681817 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:14.681786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:14.682886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:14.682863    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:16.482826 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.482791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:02:16.485376 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.485358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.487595 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.487569    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-904baa\""
Jul 28 15:02:16.487734 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.487569    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-904baa-dockercfg-2wgnp\""
Jul 28 15:02:16.493450 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.493427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:02:16.510099 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.510059    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:02:16.518345 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.518312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.520669 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.520640    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:02:16.586626 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.586587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzs8h\" (UniqueName: \"kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.586924 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.586896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.587089 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.587054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.587222 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.587105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzl5x\" (UniqueName: \"kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.688349 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.688312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzs8h\" (UniqueName: \"kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.688526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.688358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.688526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.688382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.688526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.688405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzl5x\" (UniqueName: \"kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.689137 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.689116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.689137 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.689126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.696024 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.695998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzl5x\" (UniqueName: \"kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x\") pod \"payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") " pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.696158 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.696048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzs8h\" (UniqueName: \"kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h\") pod \"payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.800478 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.800386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:16.830349 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.830317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:16.986037 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:16.985924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:02:16.988341 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:02:16.988314    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233031d0_e3bb_42f8_a9aa_89f116361259.slice/crio-f21331e7456bdd59f35e7263df208434c585e88e588c9a4028ffa7d68b7a55ef WatchSource:0}: Error finding container f21331e7456bdd59f35e7263df208434c585e88e588c9a4028ffa7d68b7a55ef: Status 404 returned error can't find the container with id f21331e7456bdd59f35e7263df208434c585e88e588c9a4028ffa7d68b7a55ef
Jul 28 15:02:17.162815 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.162789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:02:17.164601 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:02:17.164574    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d35bdc8_64c7_4eae_9327_6d3fa99a0946.slice/crio-6f8fb65237f6fbfc1fb77341bf84bc8f37f426a03627e1f9563caee391cba6b5 WatchSource:0}: Error finding container 6f8fb65237f6fbfc1fb77341bf84bc8f37f426a03627e1f9563caee391cba6b5: Status 404 returned error can't find the container with id 6f8fb65237f6fbfc1fb77341bf84bc8f37f426a03627e1f9563caee391cba6b5
Jul 28 15:02:17.700473 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.700435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" event={"ID":"233031d0-e3bb-42f8-a9aa-89f116361259","Type":"ContainerStarted","Data":"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"}
Jul 28 15:02:17.701050 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.700490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" event={"ID":"233031d0-e3bb-42f8-a9aa-89f116361259","Type":"ContainerStarted","Data":"f21331e7456bdd59f35e7263df208434c585e88e588c9a4028ffa7d68b7a55ef"}
Jul 28 15:02:17.701050 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.700518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:17.702066 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.702038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" event={"ID":"2d35bdc8-64c7-4eae-9327-6d3fa99a0946","Type":"ContainerStarted","Data":"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"}
Jul 28 15:02:17.702198 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.702097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:17.702198 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.702114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" event={"ID":"2d35bdc8-64c7-4eae-9327-6d3fa99a0946","Type":"ContainerStarted","Data":"6f8fb65237f6fbfc1fb77341bf84bc8f37f426a03627e1f9563caee391cba6b5"}
Jul 28 15:02:17.715390 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.715344    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" podStartSLOduration=1.715330438 podStartE2EDuration="1.715330438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:02:17.714254184 +0000 UTC m=+2375.400332294" watchObservedRunningTime="2026-07-28 15:02:17.715330438 +0000 UTC m=+2375.401408550"
Jul 28 15:02:17.728654 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:17.728602    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" podStartSLOduration=1.728581491 podStartE2EDuration="1.728581491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:02:17.727376942 +0000 UTC m=+2375.413455057" watchObservedRunningTime="2026-07-28 15:02:17.728581491 +0000 UTC m=+2375.414659607"
Jul 28 15:02:28.710298 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:28.710262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:02:28.710840 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:28.710481    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:02:30.723783 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:30.723728    2577 patch_prober.go:28] interesting pod/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 15:02:30.724202 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:30.723844    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 15:02:31.775149 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.775113    2577 generic.go:356] "Generic (PLEG): container finished" podID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerID="a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64" exitCode=0
Jul 28 15:02:31.775614 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.775154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerDied","Data":"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"}
Jul 28 15:02:31.775614 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.775562    2577 scope.go:117] "RemoveContainer" containerID="a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"
Jul 28 15:02:31.776608 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.776584    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:31.776670 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.776648    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:31.776710 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:31.776681    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:02:32.785895 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:32.785858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerStarted","Data":"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"}
Jul 28 15:02:33.616842 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:33.616805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:33.617275 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:33.617153    2577 patch_prober.go:28] interesting pod/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 28 15:02:33.617445 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:33.617420    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 28 15:02:34.617173 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:34.616690    2577 patch_prober.go:28] interesting pod/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 28 15:02:34.617173 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:34.616773    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 28 15:02:35.616661 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:35.616616    2577 patch_prober.go:28] interesting pod/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 28 15:02:35.616893 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:35.616686    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 28 15:02:36.616885 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:36.616840    2577 patch_prober.go:28] interesting pod/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 28 15:02:36.617278 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:36.616914    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 28 15:02:37.619844 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:37.619810    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:37.620303 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:37.620082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:37.620728 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:37.620710    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:02:42.892924 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:42.892896    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:02:42.897918 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:42.897882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:02:42.901418 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:42.901394    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:02:42.904648 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:02:42.904627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:03:51.587904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.587862    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"]
Jul 28 15:03:51.591556 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.591523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:51.593776 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.593749    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:03:51.598583 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.598560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"]
Jul 28 15:03:51.713192 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.713155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst\" (UID: \"3fa9138c-e544-4c97-86d6-7c694fc75a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:51.814240 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.814206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst\" (UID: \"3fa9138c-e544-4c97-86d6-7c694fc75a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:51.814877 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.814859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst\" (UID: \"3fa9138c-e544-4c97-86d6-7c694fc75a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:51.906378 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:51.906282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:52.043400 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:52.043373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"]
Jul 28 15:03:52.045635 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:03:52.045607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa9138c_e544_4c97_86d6_7c694fc75a1a.slice/crio-3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0 WatchSource:0}: Error finding container 3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0: Status 404 returned error can't find the container with id 3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0
Jul 28 15:03:52.179524 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:52.179439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst" event={"ID":"3fa9138c-e544-4c97-86d6-7c694fc75a1a","Type":"ContainerStarted","Data":"3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0"}
Jul 28 15:03:53.185669 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:53.185574    2577 generic.go:356] "Generic (PLEG): container finished" podID="3fa9138c-e544-4c97-86d6-7c694fc75a1a" containerID="bb4c40d0c706f19890b3fed631bdb3387fc968aa7a90a32e83da3aca98c85c33" exitCode=0
Jul 28 15:03:53.186061 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:53.185669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst" event={"ID":"3fa9138c-e544-4c97-86d6-7c694fc75a1a","Type":"ContainerDied","Data":"bb4c40d0c706f19890b3fed631bdb3387fc968aa7a90a32e83da3aca98c85c33"}
Jul 28 15:03:54.326518 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:54.326495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:54.438101 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:54.438064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca\") pod \"3fa9138c-e544-4c97-86d6-7c694fc75a1a\" (UID: \"3fa9138c-e544-4c97-86d6-7c694fc75a1a\") "
Jul 28 15:03:54.438444 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:54.438420    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fa9138c-e544-4c97-86d6-7c694fc75a1a" (UID: "3fa9138c-e544-4c97-86d6-7c694fc75a1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:03:54.539173 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:54.539139    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9138c-e544-4c97-86d6-7c694fc75a1a-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:03:55.196790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:55.196712    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"
Jul 28 15:03:55.196790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:55.196742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst" event={"ID":"3fa9138c-e544-4c97-86d6-7c694fc75a1a","Type":"ContainerDied","Data":"3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0"}
Jul 28 15:03:55.196790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:03:55.196787    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3df3b26055c760a7c028cf8be6ca7e9a17297ed56ca7a4f9c6ae8e35d0696fb0"
Jul 28 15:04:01.967725 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:01.967685    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:04:01.968337 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:01.967991    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" podUID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" containerName="maas-api" containerID="cri-o://4753472b77f1101ce0e50aacf219e12480c4fc8dc2c386b5d5d682a37656ec1c" gracePeriod=30
Jul 28 15:04:02.011819 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.011777    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:04:02.012134 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.012074    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" podUID="dbcb9e13-9443-4beb-8338-fe39daaf2230" containerName="payload-processing" containerID="cri-o://f42c696c29c990c5eb0519a411d82093ab202b97540255a01d8cab9158625f1a" gracePeriod=30
Jul 28 15:04:02.016291 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.016257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:04:02.016711 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.016492    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" podUID="24508d7a-c673-44d7-83ad-6e541b768907" containerName="payload-pre-processing" containerID="cri-o://36778335ed71e187f8c915d9e77824beb8c1cf9030abb722994f581bed398000" gracePeriod=30
Jul 28 15:04:02.153011 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.152949    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"]
Jul 28 15:04:02.163319 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.159793    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d392e9-fkdst"]
Jul 28 15:04:02.233754 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.233647    2577 generic.go:356] "Generic (PLEG): container finished" podID="24508d7a-c673-44d7-83ad-6e541b768907" containerID="36778335ed71e187f8c915d9e77824beb8c1cf9030abb722994f581bed398000" exitCode=0
Jul 28 15:04:02.233754 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.233740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" event={"ID":"24508d7a-c673-44d7-83ad-6e541b768907","Type":"ContainerDied","Data":"36778335ed71e187f8c915d9e77824beb8c1cf9030abb722994f581bed398000"}
Jul 28 15:04:02.235856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.235831    2577 generic.go:356] "Generic (PLEG): container finished" podID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" containerID="4753472b77f1101ce0e50aacf219e12480c4fc8dc2c386b5d5d682a37656ec1c" exitCode=0
Jul 28 15:04:02.236101 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.235884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" event={"ID":"3636bab2-2310-4f8e-a03b-a9b67e3091ec","Type":"ContainerDied","Data":"4753472b77f1101ce0e50aacf219e12480c4fc8dc2c386b5d5d682a37656ec1c"}
Jul 28 15:04:02.237935 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.237913    2577 generic.go:356] "Generic (PLEG): container finished" podID="dbcb9e13-9443-4beb-8338-fe39daaf2230" containerID="f42c696c29c990c5eb0519a411d82093ab202b97540255a01d8cab9158625f1a" exitCode=0
Jul 28 15:04:02.238068 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.237989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" event={"ID":"dbcb9e13-9443-4beb-8338-fe39daaf2230","Type":"ContainerDied","Data":"f42c696c29c990c5eb0519a411d82093ab202b97540255a01d8cab9158625f1a"}
Jul 28 15:04:02.300627 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.300599    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:04:02.342088 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.342058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:04:02.353967 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.353943    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:04:02.402004 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.401968    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca\") pod \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") "
Jul 28 15:04:02.402004 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume\") pod \"24508d7a-c673-44d7-83ad-6e541b768907\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") "
Jul 28 15:04:02.402250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume\") pod \"dbcb9e13-9443-4beb-8338-fe39daaf2230\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") "
Jul 28 15:04:02.402250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhbvv\" (UniqueName: \"kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv\") pod \"dbcb9e13-9443-4beb-8338-fe39daaf2230\" (UID: \"dbcb9e13-9443-4beb-8338-fe39daaf2230\") "
Jul 28 15:04:02.402250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") pod \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") "
Jul 28 15:04:02.402407 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm\") pod \"24508d7a-c673-44d7-83ad-6e541b768907\" (UID: \"24508d7a-c673-44d7-83ad-6e541b768907\") "
Jul 28 15:04:02.402407 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk9p7\" (UniqueName: \"kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7\") pod \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\" (UID: \"3636bab2-2310-4f8e-a03b-a9b67e3091ec\") "
Jul 28 15:04:02.402527 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3636bab2-2310-4f8e-a03b-a9b67e3091ec" (UID: "3636bab2-2310-4f8e-a03b-a9b67e3091ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:02.402527 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbcb9e13-9443-4beb-8338-fe39daaf2230" (UID: "dbcb9e13-9443-4beb-8338-fe39daaf2230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:02.402527 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402498    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24508d7a-c673-44d7-83ad-6e541b768907" (UID: "24508d7a-c673-44d7-83ad-6e541b768907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:02.402787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402758    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3636bab2-2310-4f8e-a03b-a9b67e3091ec-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.402787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402786    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24508d7a-c673-44d7-83ad-6e541b768907-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.402982 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.402801    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcb9e13-9443-4beb-8338-fe39daaf2230-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.404400 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.404372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3636bab2-2310-4f8e-a03b-a9b67e3091ec" (UID: "3636bab2-2310-4f8e-a03b-a9b67e3091ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:04:02.404400 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.404371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv" (OuterVolumeSpecName: "kube-api-access-rhbvv") pod "dbcb9e13-9443-4beb-8338-fe39daaf2230" (UID: "dbcb9e13-9443-4beb-8338-fe39daaf2230"). InnerVolumeSpecName "kube-api-access-rhbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:02.404558 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.404417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm" (OuterVolumeSpecName: "kube-api-access-vkwdm") pod "24508d7a-c673-44d7-83ad-6e541b768907" (UID: "24508d7a-c673-44d7-83ad-6e541b768907"). InnerVolumeSpecName "kube-api-access-vkwdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:02.404558 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.404445    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7" (OuterVolumeSpecName: "kube-api-access-qk9p7") pod "3636bab2-2310-4f8e-a03b-a9b67e3091ec" (UID: "3636bab2-2310-4f8e-a03b-a9b67e3091ec"). InnerVolumeSpecName "kube-api-access-qk9p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:02.503970 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.503872    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/24508d7a-c673-44d7-83ad-6e541b768907-kube-api-access-vkwdm\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.503970 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.503908    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk9p7\" (UniqueName: \"kubernetes.io/projected/3636bab2-2310-4f8e-a03b-a9b67e3091ec-kube-api-access-qk9p7\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.503970 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.503918    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbvv\" (UniqueName: \"kubernetes.io/projected/dbcb9e13-9443-4beb-8338-fe39daaf2230-kube-api-access-rhbvv\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.503970 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.503927    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3636bab2-2310-4f8e-a03b-a9b67e3091ec-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:02.786810 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:02.786713    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa9138c-e544-4c97-86d6-7c694fc75a1a" path="/var/lib/kubelet/pods/3fa9138c-e544-4c97-86d6-7c694fc75a1a/volumes"
Jul 28 15:04:03.245839 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.245812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"
Jul 28 15:04:03.246352 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.245852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4" event={"ID":"dbcb9e13-9443-4beb-8338-fe39daaf2230","Type":"ContainerDied","Data":"eae3ef556595c582d0df1b86e6c6a50cf5e28b3ef01581d0c7aaf4ff0e864b5f"}
Jul 28 15:04:03.246352 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.245923    2577 scope.go:117] "RemoveContainer" containerID="f42c696c29c990c5eb0519a411d82093ab202b97540255a01d8cab9158625f1a"
Jul 28 15:04:03.247245 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.247219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"
Jul 28 15:04:03.247333 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.247243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244" event={"ID":"24508d7a-c673-44d7-83ad-6e541b768907","Type":"ContainerDied","Data":"28e8af7bdc104178b9814a9e5e2ea6f26cbc9b17ec1d3c88604e5716176ff099"}
Jul 28 15:04:03.248776 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.248756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj" event={"ID":"3636bab2-2310-4f8e-a03b-a9b67e3091ec","Type":"ContainerDied","Data":"002158d0f1efac3275141302867054902686d802b25680ec7d445bed030c7ca0"}
Jul 28 15:04:03.248868 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.248797    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"
Jul 28 15:04:03.255688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.255668    2577 scope.go:117] "RemoveContainer" containerID="36778335ed71e187f8c915d9e77824beb8c1cf9030abb722994f581bed398000"
Jul 28 15:04:03.263735 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.263715    2577 scope.go:117] "RemoveContainer" containerID="4753472b77f1101ce0e50aacf219e12480c4fc8dc2c386b5d5d682a37656ec1c"
Jul 28 15:04:03.268752 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.268729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:04:03.272166 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.272132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d392e9-69678465c4-scdz4"]
Jul 28 15:04:03.282047 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.282007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:04:03.283759 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.283739    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d392e9-7b9b47b744-dg244"]
Jul 28 15:04:03.296334 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.296307    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:04:03.299583 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:03.299530    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d392e9-6c4cd9db-46szj"]
Jul 28 15:04:04.786394 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:04.786359    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24508d7a-c673-44d7-83ad-6e541b768907" path="/var/lib/kubelet/pods/24508d7a-c673-44d7-83ad-6e541b768907/volumes"
Jul 28 15:04:04.786807 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:04.786781    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" path="/var/lib/kubelet/pods/3636bab2-2310-4f8e-a03b-a9b67e3091ec/volumes"
Jul 28 15:04:04.787096 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:04.787083    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbcb9e13-9443-4beb-8338-fe39daaf2230" path="/var/lib/kubelet/pods/dbcb9e13-9443-4beb-8338-fe39daaf2230/volumes"
Jul 28 15:04:08.740136 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:08.740103    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:04:08.740700 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:08.740392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy" containerID="cri-o://3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901" gracePeriod=30
Jul 28 15:04:13.992621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:13.992593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:04:14.113196 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113196 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113152    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113196 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp2d6\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113434 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data\") pod \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\" (UID: \"940fbf67-d361-47b7-b8a1-ef84aa6387d6\") "
Jul 28 15:04:14.113792 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113603    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:14.113792 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:14.113920 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:14.113920 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113836    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.113920 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.113856    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.114169 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.114145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:14.114680 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.114654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:14.115608 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.115574    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6" (OuterVolumeSpecName: "kube-api-access-mp2d6") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "kube-api-access-mp2d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:14.115608 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.115594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:04:14.115749 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.115590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:14.115749 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.115639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "940fbf67-d361-47b7-b8a1-ef84aa6387d6" (UID: "940fbf67-d361-47b7-b8a1-ef84aa6387d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:14.214918 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214881    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2d6\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-kube-api-access-mp2d6\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.214918 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214912    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.214918 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214924    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.215160 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214933    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.215160 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214941    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.215160 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214949    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.215160 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.214956    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940fbf67-d361-47b7-b8a1-ef84aa6387d6-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:14.307454 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.307420    2577 generic.go:356] "Generic (PLEG): container finished" podID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerID="3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901" exitCode=0
Jul 28 15:04:14.307666 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.307475    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"
Jul 28 15:04:14.307666 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.307507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerDied","Data":"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"}
Jul 28 15:04:14.307666 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.307580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd" event={"ID":"940fbf67-d361-47b7-b8a1-ef84aa6387d6","Type":"ContainerDied","Data":"10a6179f02e7b1d9a2750109530dbeab932930057f56f43089ba0bf0f3df1377"}
Jul 28 15:04:14.307666 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.307609    2577 scope.go:117] "RemoveContainer" containerID="3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"
Jul 28 15:04:14.316733 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.316711    2577 scope.go:117] "RemoveContainer" containerID="599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"
Jul 28 15:04:14.324893 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.324873    2577 scope.go:117] "RemoveContainer" containerID="3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"
Jul 28 15:04:14.325175 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:14.325155    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901\": container with ID starting with 3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901 not found: ID does not exist" containerID="3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"
Jul 28 15:04:14.325226 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.325185    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901"} err="failed to get container status \"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901\": rpc error: code = NotFound desc = could not find container \"3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901\": container with ID starting with 3b54e864b7feb4421bb5534418005c0b30929dd33b561d2c1a0513bab254b901 not found: ID does not exist"
Jul 28 15:04:14.325226 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.325203    2577 scope.go:117] "RemoveContainer" containerID="599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"
Jul 28 15:04:14.325495 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:14.325458    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67\": container with ID starting with 599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67 not found: ID does not exist" containerID="599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"
Jul 28 15:04:14.325653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.325495    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67"} err="failed to get container status \"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67\": rpc error: code = NotFound desc = could not find container \"599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67\": container with ID starting with 599d86628c2690a6745c636f3aebd8e03e909d0e38f74687b2fc039091ad3c67 not found: ID does not exist"
Jul 28 15:04:14.329154 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.329129    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:04:14.331474 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.331454    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d392e9-openshift-default-7dfdbfd998-5j2bd"]
Jul 28 15:04:14.786126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:14.786092    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" path="/var/lib/kubelet/pods/940fbf67-d361-47b7-b8a1-ef84aa6387d6/volumes"
Jul 28 15:04:19.682139 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.682080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:04:19.682687 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.682342    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" podUID="233031d0-e3bb-42f8-a9aa-89f116361259" containerName="payload-processing" containerID="cri-o://0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258" gracePeriod=30
Jul 28 15:04:19.689522 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.689495    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:04:19.689753 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.689728    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" podUID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" containerName="payload-pre-processing" containerID="cri-o://536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e" gracePeriod=30
Jul 28 15:04:19.971126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.971098    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:04:19.974063 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:19.974041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:04:20.067761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.067723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzs8h\" (UniqueName: \"kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h\") pod \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") "
Jul 28 15:04:20.067913 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.067808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume\") pod \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\" (UID: \"2d35bdc8-64c7-4eae-9327-6d3fa99a0946\") "
Jul 28 15:04:20.067913 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.067834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzl5x\" (UniqueName: \"kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x\") pod \"233031d0-e3bb-42f8-a9aa-89f116361259\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") "
Jul 28 15:04:20.067913 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.067902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume\") pod \"233031d0-e3bb-42f8-a9aa-89f116361259\" (UID: \"233031d0-e3bb-42f8-a9aa-89f116361259\") "
Jul 28 15:04:20.068320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.068294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d35bdc8-64c7-4eae-9327-6d3fa99a0946" (UID: "2d35bdc8-64c7-4eae-9327-6d3fa99a0946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:20.068459 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.068317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "233031d0-e3bb-42f8-a9aa-89f116361259" (UID: "233031d0-e3bb-42f8-a9aa-89f116361259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:20.069929 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.069906    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x" (OuterVolumeSpecName: "kube-api-access-lzl5x") pod "233031d0-e3bb-42f8-a9aa-89f116361259" (UID: "233031d0-e3bb-42f8-a9aa-89f116361259"). InnerVolumeSpecName "kube-api-access-lzl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:20.070008 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.069973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h" (OuterVolumeSpecName: "kube-api-access-hzs8h") pod "2d35bdc8-64c7-4eae-9327-6d3fa99a0946" (UID: "2d35bdc8-64c7-4eae-9327-6d3fa99a0946"). InnerVolumeSpecName "kube-api-access-hzs8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:20.168662 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.168623    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzs8h\" (UniqueName: \"kubernetes.io/projected/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-kube-api-access-hzs8h\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:20.168662 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.168667    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d35bdc8-64c7-4eae-9327-6d3fa99a0946-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:20.168874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.168684    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzl5x\" (UniqueName: \"kubernetes.io/projected/233031d0-e3bb-42f8-a9aa-89f116361259-kube-api-access-lzl5x\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:20.168874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.168703    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/233031d0-e3bb-42f8-a9aa-89f116361259-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:20.338447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.338415    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" containerID="536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e" exitCode=0
Jul 28 15:04:20.338674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.338474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"
Jul 28 15:04:20.338674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.338503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" event={"ID":"2d35bdc8-64c7-4eae-9327-6d3fa99a0946","Type":"ContainerDied","Data":"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"}
Jul 28 15:04:20.338674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.338583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx" event={"ID":"2d35bdc8-64c7-4eae-9327-6d3fa99a0946","Type":"ContainerDied","Data":"6f8fb65237f6fbfc1fb77341bf84bc8f37f426a03627e1f9563caee391cba6b5"}
Jul 28 15:04:20.338674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.338615    2577 scope.go:117] "RemoveContainer" containerID="536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"
Jul 28 15:04:20.339977 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.339957    2577 generic.go:356] "Generic (PLEG): container finished" podID="233031d0-e3bb-42f8-a9aa-89f116361259" containerID="0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258" exitCode=0
Jul 28 15:04:20.340096 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.339984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" event={"ID":"233031d0-e3bb-42f8-a9aa-89f116361259","Type":"ContainerDied","Data":"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"}
Jul 28 15:04:20.340096 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.340005    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"
Jul 28 15:04:20.340096 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.340038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7" event={"ID":"233031d0-e3bb-42f8-a9aa-89f116361259","Type":"ContainerDied","Data":"f21331e7456bdd59f35e7263df208434c585e88e588c9a4028ffa7d68b7a55ef"}
Jul 28 15:04:20.349185 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.349163    2577 scope.go:117] "RemoveContainer" containerID="536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"
Jul 28 15:04:20.349499 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:20.349473    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e\": container with ID starting with 536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e not found: ID does not exist" containerID="536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"
Jul 28 15:04:20.349612 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.349513    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e"} err="failed to get container status \"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e\": rpc error: code = NotFound desc = could not find container \"536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e\": container with ID starting with 536fa351eaff968704e5f63641aecaf8c4376bc67da69fcf46afe5613f4ce79e not found: ID does not exist"
Jul 28 15:04:20.349612 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.349562    2577 scope.go:117] "RemoveContainer" containerID="0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"
Jul 28 15:04:20.357629 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.357611    2577 scope.go:117] "RemoveContainer" containerID="0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"
Jul 28 15:04:20.357915 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:20.357888    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258\": container with ID starting with 0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258 not found: ID does not exist" containerID="0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"
Jul 28 15:04:20.357963 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.357923    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258"} err="failed to get container status \"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258\": rpc error: code = NotFound desc = could not find container \"0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258\": container with ID starting with 0f54cdf50071f15164c742e24dc80b51975174782cdf390115a51ea4b4a6f258 not found: ID does not exist"
Jul 28 15:04:20.362442 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.362417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:04:20.366976 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.366950    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-904baa-68f7b75cd8-ltfcx"]
Jul 28 15:04:20.375287 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.375266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:04:20.378679 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.378658    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-904baa-647d5c89d6-pthp7"]
Jul 28 15:04:20.785509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.785479    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233031d0-e3bb-42f8-a9aa-89f116361259" path="/var/lib/kubelet/pods/233031d0-e3bb-42f8-a9aa-89f116361259/volumes"
Jul 28 15:04:20.785904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:20.785847    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" path="/var/lib/kubelet/pods/2d35bdc8-64c7-4eae-9327-6d3fa99a0946/volumes"
Jul 28 15:04:26.392204 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:26.392164    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:04:26.392651 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:26.392409    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy" containerID="cri-o://576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904" gracePeriod=30
Jul 28 15:04:27.429660 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.429623    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.429987    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcb9e13-9443-4beb-8338-fe39daaf2230" containerName="payload-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.429999    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcb9e13-9443-4beb-8338-fe39daaf2230" containerName="payload-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430011    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233031d0-e3bb-42f8-a9aa-89f116361259" containerName="payload-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430017    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="233031d0-e3bb-42f8-a9aa-89f116361259" containerName="payload-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430028    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" containerName="payload-pre-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430034    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" containerName="payload-pre-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430045    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430050    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430056    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa9138c-e544-4c97-86d6-7c694fc75a1a" containerName="revoke-keys"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430062    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa9138c-e544-4c97-86d6-7c694fc75a1a" containerName="revoke-keys"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430072    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430077    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430086    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24508d7a-c673-44d7-83ad-6e541b768907" containerName="payload-pre-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430091    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24508d7a-c673-44d7-83ad-6e541b768907" containerName="payload-pre-processing"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430098    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" containerName="maas-api"
Jul 28 15:04:27.430126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430103    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" containerName="maas-api"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430155    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3636bab2-2310-4f8e-a03b-a9b67e3091ec" containerName="maas-api"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430167    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa9138c-e544-4c97-86d6-7c694fc75a1a" containerName="revoke-keys"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430177    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430183    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="233031d0-e3bb-42f8-a9aa-89f116361259" containerName="payload-processing"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430188    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24508d7a-c673-44d7-83ad-6e541b768907" containerName="payload-pre-processing"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d35bdc8-64c7-4eae-9327-6d3fa99a0946" containerName="payload-pre-processing"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430199    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcb9e13-9443-4beb-8338-fe39daaf2230" containerName="payload-processing"
Jul 28 15:04:27.430947 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.430204    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="940fbf67-d361-47b7-b8a1-ef84aa6387d6" containerName="istio-proxy"
Jul 28 15:04:27.434594 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.434574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.436805 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.436786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7d80d5-openshift-default-dockercfg-gmvm5\""
Jul 28 15:04:27.445199 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.445176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:04:27.533193 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fcg\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533377 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.533659 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.533415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634762 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.634957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635169 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.634989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635169 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635169 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fcg\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635317 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.635857 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.635833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.637315 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.637293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.637451 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.637434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.641935 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.641914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.642147 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.642125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fcg\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg\") pod \"e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") " pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.747180 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.747142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:27.883235 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.883136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:04:27.885564 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:04:27.885516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dec8dbb_3dd5_4a1f_9679_11f8fbf0e490.slice/crio-a4fda60a59e751dfae6ba008179f50e62c8e21d475b382ccedc0a0ccca0965ca WatchSource:0}: Error finding container a4fda60a59e751dfae6ba008179f50e62c8e21d475b382ccedc0a0ccca0965ca: Status 404 returned error can't find the container with id a4fda60a59e751dfae6ba008179f50e62c8e21d475b382ccedc0a0ccca0965ca
Jul 28 15:04:27.887760 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.887728    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:27.887858 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.887792    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:27.887858 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:27.887819    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:28.390765 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:28.390731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerStarted","Data":"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"}
Jul 28 15:04:28.390949 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:28.390785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerStarted","Data":"a4fda60a59e751dfae6ba008179f50e62c8e21d475b382ccedc0a0ccca0965ca"}
Jul 28 15:04:28.411505 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:28.411455    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podStartSLOduration=1.411438534 podStartE2EDuration="1.411438534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:04:28.408874328 +0000 UTC m=+2506.094952440" watchObservedRunningTime="2026-07-28 15:04:28.411438534 +0000 UTC m=+2506.097516647"
Jul 28 15:04:28.748131 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:28.748093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:28.753208 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:28.753181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:29.395475 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:29.395442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:29.396701 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:29.396670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:30.434974 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.434940    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:04:30.438949 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.438927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.441002 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.440981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7d80d5-serving-cert\""
Jul 28 15:04:30.441122 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.441078    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 15:04:30.446803 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.446778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:04:30.562078 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.562042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.562300 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.562095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.562300 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.562170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5sr\" (UniqueName: \"kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.663589 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.663548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5sr\" (UniqueName: \"kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.663772 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.663672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.663772 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.663727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.663863 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:30.663847    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-serving-cert: secret "maas-api-e2e-api-a-7d80d5-serving-cert" not found
Jul 28 15:04:30.663936 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:30.663926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls podName:49736725-8b0c-4686-8f48-22c1779f1a1c nodeName:}" failed. No retries permitted until 2026-07-28 15:04:31.163904886 +0000 UTC m=+2508.849982989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls") pod "maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" (UID: "49736725-8b0c-4686-8f48-22c1779f1a1c") : secret "maas-api-e2e-api-a-7d80d5-serving-cert" not found
Jul 28 15:04:30.664304 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.664287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:30.672118 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:30.672087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5sr\" (UniqueName: \"kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:31.023700 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.023661    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:04:31.027363 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.027341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.029447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.029419    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7d80d5\""
Jul 28 15:04:31.029598 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.029480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7d80d5-dockercfg-hwnzd\""
Jul 28 15:04:31.036473 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.036445    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:04:31.060409 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.060382    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:04:31.063871 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.063854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.073590 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.073564    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:04:31.168565 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.168502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44fh\" (UniqueName: \"kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.168768 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.168604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:31.168768 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.168635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.168768 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.168698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvg5\" (UniqueName: \"kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.168768 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.168722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.171206 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.171177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") pod \"maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:31.270012 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.269977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44fh\" (UniqueName: \"kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.270201 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.270026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.270201 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.270069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvg5\" (UniqueName: \"kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.270201 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.270086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.270733 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.270713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.270787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.270775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.277693 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.277621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvg5\" (UniqueName: \"kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5\") pod \"payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") " pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.277693 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.277636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44fh\" (UniqueName: \"kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh\") pod \"payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.339549 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.339505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:31.352846 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.352823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:31.374404 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.374370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:31.502618 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.502509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:04:31.504298 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:04:31.504239    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb492af26_8598_46fb_b6bc_d22159cbb47a.slice/crio-c9744fffc7cdff2806853ffc8625b46c156939f845ba1f1d466738b86f8c562a WatchSource:0}: Error finding container c9744fffc7cdff2806853ffc8625b46c156939f845ba1f1d466738b86f8c562a: Status 404 returned error can't find the container with id c9744fffc7cdff2806853ffc8625b46c156939f845ba1f1d466738b86f8c562a
Jul 28 15:04:31.540468 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.540135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:04:31.548164 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:04:31.547942    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49736725_8b0c_4686_8f48_22c1779f1a1c.slice/crio-a373bf4a91d678186ba5071f6ed29502633d058cc144d61ee74a8bd004cfe498 WatchSource:0}: Error finding container a373bf4a91d678186ba5071f6ed29502633d058cc144d61ee74a8bd004cfe498: Status 404 returned error can't find the container with id a373bf4a91d678186ba5071f6ed29502633d058cc144d61ee74a8bd004cfe498
Jul 28 15:04:31.550134 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.550111    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:04:31.574785 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.574732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:04:31.577636 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:04:31.577608    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fffb50a_6c37_44e3_950c_ce2ceaa03dc3.slice/crio-b45b756ef750741fd1dd595d5bc7059cefbd81665a6fcca7948e5d7efe3f8b15 WatchSource:0}: Error finding container b45b756ef750741fd1dd595d5bc7059cefbd81665a6fcca7948e5d7efe3f8b15: Status 404 returned error can't find the container with id b45b756ef750741fd1dd595d5bc7059cefbd81665a6fcca7948e5d7efe3f8b15
Jul 28 15:04:31.655107 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.655083    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:04:31.775627 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7xm4\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.775793 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.775793 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.775793 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.775977 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.775977 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775947    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.776083 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.775981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.776083 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.776083 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket\") pod \"e62269da-cd45-481d-8d2c-98e27d238fb8\" (UID: \"e62269da-cd45-481d-8d2c-98e27d238fb8\") "
Jul 28 15:04:31.776248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:31.776248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:31.776676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776398    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.776676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:31.776676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776422    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.776676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:31.776994 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.776920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:31.778091 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.778066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:31.778091 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.778074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:31.778248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.778224    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:04:31.778338 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.778326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4" (OuterVolumeSpecName: "kube-api-access-l7xm4") pod "e62269da-cd45-481d-8d2c-98e27d238fb8" (UID: "e62269da-cd45-481d-8d2c-98e27d238fb8"). InnerVolumeSpecName "kube-api-access-l7xm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:31.877688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877645    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877679    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877689    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7xm4\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-kube-api-access-l7xm4\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877698    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877956 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877706    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877956 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877715    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62269da-cd45-481d-8d2c-98e27d238fb8-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:31.877956 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:31.877722    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62269da-cd45-481d-8d2c-98e27d238fb8-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:04:32.415794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.415696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" event={"ID":"49736725-8b0c-4686-8f48-22c1779f1a1c","Type":"ContainerStarted","Data":"3bb11a60533fd705f5b1dd8090b85c3e59213e02c1b0e93b4db3c9de897a5109"}
Jul 28 15:04:32.415794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.415751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" event={"ID":"49736725-8b0c-4686-8f48-22c1779f1a1c","Type":"ContainerStarted","Data":"a373bf4a91d678186ba5071f6ed29502633d058cc144d61ee74a8bd004cfe498"}
Jul 28 15:04:32.416049 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.415834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:32.417509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.417484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" event={"ID":"b492af26-8598-46fb-b6bc-d22159cbb47a","Type":"ContainerStarted","Data":"4d00262d6b802acaca8eb843123e59941f6b10aad5d9b36a44c2f50b775b4591"}
Jul 28 15:04:32.417658 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.417554    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:32.417658 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.417580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" event={"ID":"b492af26-8598-46fb-b6bc-d22159cbb47a","Type":"ContainerStarted","Data":"c9744fffc7cdff2806853ffc8625b46c156939f845ba1f1d466738b86f8c562a"}
Jul 28 15:04:32.419305 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.419285    2577 generic.go:356] "Generic (PLEG): container finished" podID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerID="576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904" exitCode=0
Jul 28 15:04:32.419406 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.419314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerDied","Data":"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"}
Jul 28 15:04:32.419406 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.419326    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"
Jul 28 15:04:32.419406 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.419362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7" event={"ID":"e62269da-cd45-481d-8d2c-98e27d238fb8","Type":"ContainerDied","Data":"184204165f30ba58041b1c4b85f49f063c93747c63b7d6f473c069d68873460c"}
Jul 28 15:04:32.419406 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.419391    2577 scope.go:117] "RemoveContainer" containerID="576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"
Jul 28 15:04:32.421020 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.420997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" event={"ID":"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3","Type":"ContainerStarted","Data":"9467f58cc0ea004e6bc1ef9665106bd79a7ae86f49d889d089d5f2114dc1bb41"}
Jul 28 15:04:32.421138 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.421039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" event={"ID":"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3","Type":"ContainerStarted","Data":"b45b756ef750741fd1dd595d5bc7059cefbd81665a6fcca7948e5d7efe3f8b15"}
Jul 28 15:04:32.421138 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.421109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:32.430795 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.430778    2577 scope.go:117] "RemoveContainer" containerID="a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"
Jul 28 15:04:32.432041 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.432007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" podStartSLOduration=1.8637443999999999 podStartE2EDuration="2.431993086s" totalImagesPullingTime="568.248686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:04:32.43068541 +0000 UTC m=+2510.116763516" watchObservedRunningTime="2026-07-28 15:04:32.431993086 +0000 UTC m=+2510.118071191"
Jul 28 15:04:32.440357 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.440331    2577 scope.go:117] "RemoveContainer" containerID="576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"
Jul 28 15:04:32.440780 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:32.440753    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904\": container with ID starting with 576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904 not found: ID does not exist" containerID="576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"
Jul 28 15:04:32.440884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.440792    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904"} err="failed to get container status \"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904\": rpc error: code = NotFound desc = could not find container \"576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904\": container with ID starting with 576d95a7ac8550b80fc2acf3cc736ae3bd09054e62ed6cd53ad441159ed0b904 not found: ID does not exist"
Jul 28 15:04:32.440884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.440817    2577 scope.go:117] "RemoveContainer" containerID="a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"
Jul 28 15:04:32.441110 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:04:32.441087    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64\": container with ID starting with a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64 not found: ID does not exist" containerID="a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"
Jul 28 15:04:32.441171 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.441117    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64"} err="failed to get container status \"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64\": rpc error: code = NotFound desc = could not find container \"a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64\": container with ID starting with a0b9326df235f01043ad106bfbcd9d288d05a50109eb2b98bd3005dc0e659b64 not found: ID does not exist"
Jul 28 15:04:32.449245 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.449181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:04:32.451891 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.451863    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-904baa-openshift-default-59885b7858-c44r7"]
Jul 28 15:04:32.464300 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.464250    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" podStartSLOduration=1.464235354 podStartE2EDuration="1.464235354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:04:32.46297527 +0000 UTC m=+2510.149053384" watchObservedRunningTime="2026-07-28 15:04:32.464235354 +0000 UTC m=+2510.150313466"
Jul 28 15:04:32.478462 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.478413    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" podStartSLOduration=1.478399092 podStartE2EDuration="1.478399092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:04:32.477040544 +0000 UTC m=+2510.163118658" watchObservedRunningTime="2026-07-28 15:04:32.478399092 +0000 UTC m=+2510.164477206"
Jul 28 15:04:32.790690 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:32.790657    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" path="/var/lib/kubelet/pods/e62269da-cd45-481d-8d2c-98e27d238fb8/volumes"
Jul 28 15:04:38.436236 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:38.436151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:04:43.430152 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.430043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:04:43.430152 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.430117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:04:43.482416 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.482387    2577 generic.go:356] "Generic (PLEG): container finished" podID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerID="7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79" exitCode=0
Jul 28 15:04:43.482641 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.482455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerDied","Data":"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"}
Jul 28 15:04:43.482893 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.482877    2577 scope.go:117] "RemoveContainer" containerID="7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"
Jul 28 15:04:43.484111 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.484080    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:43.484196 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.484167    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:43.484241 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:43.484215    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:44.395399 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:44.395359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:44.490600 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:44.490564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerStarted","Data":"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"}
Jul 28 15:04:44.748347 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:44.748306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:44.749640 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:44.749615    2577 patch_prober.go:28] interesting pod/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 15:04:44.749770 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:44.749662    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 15:04:45.748589 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:45.748525    2577 patch_prober.go:28] interesting pod/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 15:04:45.749016 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:45.748610    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 15:04:46.747928 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:46.747891    2577 patch_prober.go:28] interesting pod/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 15:04:46.748094 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:46.747961    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 15:04:47.748168 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:47.748129    2577 patch_prober.go:28] interesting pod/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 28 15:04:47.748603 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:47.748185    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 28 15:04:48.751807 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:48.751771    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:48.752245 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:48.752059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:48.752855 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:48.752838    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:04:56.262370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262324    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262721    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262730    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262805    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.262856 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.262813    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62269da-cd45-481d-8d2c-98e27d238fb8" containerName="istio-proxy"
Jul 28 15:04:56.266905 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.266884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.269183 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.269165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-84fa8d-openshift-default-dockercfg-k42dk\""
Jul 28 15:04:56.279843 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.279819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:04:56.305440 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305640 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305640 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305640 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dp4\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.305790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.305764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82dp4\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.406761 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407088 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407088 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407088 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.406951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407088 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.407072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407297 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.407239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407297 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.407277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.407428 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.407409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.408789 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.408763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.409048 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.409032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.416613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.416588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dp4\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.416772 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.416756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token\") pod \"e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") " pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.577597 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.577497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:56.712268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.712235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:04:56.714866 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:04:56.714840    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6163563_0de0_46b6_aa18_95773f8f9854.slice/crio-97de50031da3b82da1d9c3207ea88ca35a21c0090660ef1a226458db4359bead WatchSource:0}: Error finding container 97de50031da3b82da1d9c3207ea88ca35a21c0090660ef1a226458db4359bead: Status 404 returned error can't find the container with id 97de50031da3b82da1d9c3207ea88ca35a21c0090660ef1a226458db4359bead
Jul 28 15:04:56.719695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.718448    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:56.719695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.718622    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:56.719695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:56.718712    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:04:57.556019 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:57.555974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerStarted","Data":"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"}
Jul 28 15:04:57.556435 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:57.556040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerStarted","Data":"97de50031da3b82da1d9c3207ea88ca35a21c0090660ef1a226458db4359bead"}
Jul 28 15:04:57.575283 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:57.575229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podStartSLOduration=1.575212424 podStartE2EDuration="1.575212424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:04:57.573604641 +0000 UTC m=+2535.259682754" watchObservedRunningTime="2026-07-28 15:04:57.575212424 +0000 UTC m=+2535.261290536"
Jul 28 15:04:57.578374 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:57.578337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:57.583378 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:57.583359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:58.561942 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:58.561909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:04:58.562895 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:04:58.562873    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:05:00.325592 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.325560    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:05:00.328301 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.328281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.330336 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.330314    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-84fa8d-dockercfg-xdksq\""
Jul 28 15:05:00.330453 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.330394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-84fa8d\""
Jul 28 15:05:00.337367 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.337343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:05:00.346855 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.346824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.346975 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.346878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.448265 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.448233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.448432 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.448283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.449181 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.449156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.455480 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.455453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz\") pod \"payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.641008 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.640926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:00.775338 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:00.775312    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:05:00.777755 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:05:00.777723    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8cd5a_8e62_4685_886f_444107861fc5.slice/crio-4ab7385cbd9da2caed1a329a3dd4b67d078ae84ab48aa40e1bc6bb7ce780f072 WatchSource:0}: Error finding container 4ab7385cbd9da2caed1a329a3dd4b67d078ae84ab48aa40e1bc6bb7ce780f072: Status 404 returned error can't find the container with id 4ab7385cbd9da2caed1a329a3dd4b67d078ae84ab48aa40e1bc6bb7ce780f072
Jul 28 15:05:01.580014 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:01.579976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" event={"ID":"e4c8cd5a-8e62-4685-886f-444107861fc5","Type":"ContainerStarted","Data":"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"}
Jul 28 15:05:01.580416 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:01.580038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" event={"ID":"e4c8cd5a-8e62-4685-886f-444107861fc5","Type":"ContainerStarted","Data":"4ab7385cbd9da2caed1a329a3dd4b67d078ae84ab48aa40e1bc6bb7ce780f072"}
Jul 28 15:05:01.580416 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:01.580066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:01.595885 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:01.595831    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" podStartSLOduration=1.5958153560000001 podStartE2EDuration="1.595815356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:05:01.593755791 +0000 UTC m=+2539.279833904" watchObservedRunningTime="2026-07-28 15:05:01.595815356 +0000 UTC m=+2539.281893469"
Jul 28 15:05:11.630068 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.629985    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6163563-0de0-46b6-aa18-95773f8f9854" containerID="9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de" exitCode=0
Jul 28 15:05:11.630068 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.630056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerDied","Data":"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"}
Jul 28 15:05:11.630501 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.630489    2577 scope.go:117] "RemoveContainer" containerID="9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"
Jul 28 15:05:11.631642 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.631615    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:05:11.631718 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.631704    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:05:11.631760 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:11.631742    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:05:12.587378 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:12.587350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:05:12.641119 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:12.641084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerStarted","Data":"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"}
Jul 28 15:05:13.562729 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:13.562691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:05:13.577952 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:13.577917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:05:13.579270 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:13.579244    2577 patch_prober.go:28] interesting pod/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 28 15:05:13.579372 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:13.579291    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 28 15:05:14.578021 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:14.577988    2577 patch_prober.go:28] interesting pod/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 28 15:05:14.578417 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:14.578046    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 28 15:05:15.578266 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:15.578227    2577 patch_prober.go:28] interesting pod/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 28 15:05:15.578735 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:15.578283    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 28 15:05:16.582812 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:16.582781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:05:16.583909 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:16.583887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:05:43.556178 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.556143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:05:43.556943 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.556911    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" podUID="49736725-8b0c-4686-8f48-22c1779f1a1c" containerName="maas-api" containerID="cri-o://3bb11a60533fd705f5b1dd8090b85c3e59213e02c1b0e93b4db3c9de897a5109" gracePeriod=30
Jul 28 15:05:43.611178 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.611141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:05:43.612308 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.612257    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" podUID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" containerName="payload-processing" containerID="cri-o://9467f58cc0ea004e6bc1ef9665106bd79a7ae86f49d889d089d5f2114dc1bb41" gracePeriod=30
Jul 28 15:05:43.616447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.616412    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:05:43.618685 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.618612    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" podUID="b492af26-8598-46fb-b6bc-d22159cbb47a" containerName="payload-pre-processing" containerID="cri-o://4d00262d6b802acaca8eb843123e59941f6b10aad5d9b36a44c2f50b775b4591" gracePeriod=30
Jul 28 15:05:43.794773 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.794721    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" containerID="9467f58cc0ea004e6bc1ef9665106bd79a7ae86f49d889d089d5f2114dc1bb41" exitCode=0
Jul 28 15:05:43.794978 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.794838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" event={"ID":"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3","Type":"ContainerDied","Data":"9467f58cc0ea004e6bc1ef9665106bd79a7ae86f49d889d089d5f2114dc1bb41"}
Jul 28 15:05:43.796753 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.796722    2577 generic.go:356] "Generic (PLEG): container finished" podID="49736725-8b0c-4686-8f48-22c1779f1a1c" containerID="3bb11a60533fd705f5b1dd8090b85c3e59213e02c1b0e93b4db3c9de897a5109" exitCode=0
Jul 28 15:05:43.796887 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.796818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" event={"ID":"49736725-8b0c-4686-8f48-22c1779f1a1c","Type":"ContainerDied","Data":"3bb11a60533fd705f5b1dd8090b85c3e59213e02c1b0e93b4db3c9de897a5109"}
Jul 28 15:05:43.798522 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.798494    2577 generic.go:356] "Generic (PLEG): container finished" podID="b492af26-8598-46fb-b6bc-d22159cbb47a" containerID="4d00262d6b802acaca8eb843123e59941f6b10aad5d9b36a44c2f50b775b4591" exitCode=0
Jul 28 15:05:43.798676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.798619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" event={"ID":"b492af26-8598-46fb-b6bc-d22159cbb47a","Type":"ContainerDied","Data":"4d00262d6b802acaca8eb843123e59941f6b10aad5d9b36a44c2f50b775b4591"}
Jul 28 15:05:43.887341 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.887313    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:05:43.921970 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.921944    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:05:43.952027 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.952000    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:05:43.986232 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd5sr\" (UniqueName: \"kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr\") pod \"49736725-8b0c-4686-8f48-22c1779f1a1c\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") "
Jul 28 15:05:43.986421 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsvg5\" (UniqueName: \"kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5\") pod \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") "
Jul 28 15:05:43.986421 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume\") pod \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\" (UID: \"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3\") "
Jul 28 15:05:43.986421 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") pod \"49736725-8b0c-4686-8f48-22c1779f1a1c\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") "
Jul 28 15:05:43.986620 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986499    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca\") pod \"49736725-8b0c-4686-8f48-22c1779f1a1c\" (UID: \"49736725-8b0c-4686-8f48-22c1779f1a1c\") "
Jul 28 15:05:43.986886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" (UID: "8fffb50a-6c37-44e3-950c-ce2ceaa03dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:43.986994 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.986897    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49736725-8b0c-4686-8f48-22c1779f1a1c" (UID: "49736725-8b0c-4686-8f48-22c1779f1a1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:43.988568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.988527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49736725-8b0c-4686-8f48-22c1779f1a1c" (UID: "49736725-8b0c-4686-8f48-22c1779f1a1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:05:43.988676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.988593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5" (OuterVolumeSpecName: "kube-api-access-dsvg5") pod "8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" (UID: "8fffb50a-6c37-44e3-950c-ce2ceaa03dc3"). InnerVolumeSpecName "kube-api-access-dsvg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:05:43.988676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:43.988618    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr" (OuterVolumeSpecName: "kube-api-access-xd5sr") pod "49736725-8b0c-4686-8f48-22c1779f1a1c" (UID: "49736725-8b0c-4686-8f48-22c1779f1a1c"). InnerVolumeSpecName "kube-api-access-xd5sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:05:44.087480 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume\") pod \"b492af26-8598-46fb-b6bc-d22159cbb47a\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") "
Jul 28 15:05:44.087675 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b44fh\" (UniqueName: \"kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh\") pod \"b492af26-8598-46fb-b6bc-d22159cbb47a\" (UID: \"b492af26-8598-46fb-b6bc-d22159cbb47a\") "
Jul 28 15:05:44.087739 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087726    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49736725-8b0c-4686-8f48-22c1779f1a1c-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.087787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087743    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49736725-8b0c-4686-8f48-22c1779f1a1c-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.087787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087752    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5sr\" (UniqueName: \"kubernetes.io/projected/49736725-8b0c-4686-8f48-22c1779f1a1c-kube-api-access-xd5sr\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.087787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087761    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsvg5\" (UniqueName: \"kubernetes.io/projected/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-kube-api-access-dsvg5\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.087787 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087769    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.087920 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.087833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b492af26-8598-46fb-b6bc-d22159cbb47a" (UID: "b492af26-8598-46fb-b6bc-d22159cbb47a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:44.089575 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.089552    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh" (OuterVolumeSpecName: "kube-api-access-b44fh") pod "b492af26-8598-46fb-b6bc-d22159cbb47a" (UID: "b492af26-8598-46fb-b6bc-d22159cbb47a"). InnerVolumeSpecName "kube-api-access-b44fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:05:44.189291 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.189205    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b492af26-8598-46fb-b6bc-d22159cbb47a-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.189291 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.189232    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44fh\" (UniqueName: \"kubernetes.io/projected/b492af26-8598-46fb-b6bc-d22159cbb47a-kube-api-access-b44fh\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:44.803881 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.803856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"
Jul 28 15:05:44.804268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.803886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj" event={"ID":"b492af26-8598-46fb-b6bc-d22159cbb47a","Type":"ContainerDied","Data":"c9744fffc7cdff2806853ffc8625b46c156939f845ba1f1d466738b86f8c562a"}
Jul 28 15:05:44.804268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.803956    2577 scope.go:117] "RemoveContainer" containerID="4d00262d6b802acaca8eb843123e59941f6b10aad5d9b36a44c2f50b775b4591"
Jul 28 15:05:44.805358 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.805295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j" event={"ID":"8fffb50a-6c37-44e3-950c-ce2ceaa03dc3","Type":"ContainerDied","Data":"b45b756ef750741fd1dd595d5bc7059cefbd81665a6fcca7948e5d7efe3f8b15"}
Jul 28 15:05:44.805358 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.805318    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"
Jul 28 15:05:44.806886 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.806861    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"
Jul 28 15:05:44.807004 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.806860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4" event={"ID":"49736725-8b0c-4686-8f48-22c1779f1a1c","Type":"ContainerDied","Data":"a373bf4a91d678186ba5071f6ed29502633d058cc144d61ee74a8bd004cfe498"}
Jul 28 15:05:44.814194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.814180    2577 scope.go:117] "RemoveContainer" containerID="9467f58cc0ea004e6bc1ef9665106bd79a7ae86f49d889d089d5f2114dc1bb41"
Jul 28 15:05:44.822365 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.822318    2577 scope.go:117] "RemoveContainer" containerID="3bb11a60533fd705f5b1dd8090b85c3e59213e02c1b0e93b4db3c9de897a5109"
Jul 28 15:05:44.825630 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.825527    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:05:44.827813 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.827788    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d80d5-68ccdc8fbc-j22v4"]
Jul 28 15:05:44.836621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.836596    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:05:44.840864 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.840841    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7d80d5-687c689dcb-cb5xj"]
Jul 28 15:05:44.850566 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.850545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:05:44.854602 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:44.854585    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7d80d5-7d976db7d8-whj8j"]
Jul 28 15:05:46.786289 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:46.786259    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49736725-8b0c-4686-8f48-22c1779f1a1c" path="/var/lib/kubelet/pods/49736725-8b0c-4686-8f48-22c1779f1a1c/volumes"
Jul 28 15:05:46.786677 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:46.786655    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" path="/var/lib/kubelet/pods/8fffb50a-6c37-44e3-950c-ce2ceaa03dc3/volumes"
Jul 28 15:05:46.786988 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:46.786975    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b492af26-8598-46fb-b6bc-d22159cbb47a" path="/var/lib/kubelet/pods/b492af26-8598-46fb-b6bc-d22159cbb47a/volumes"
Jul 28 15:05:50.566469 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:50.566424    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:05:50.567129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:50.566645    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy" containerID="cri-o://56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691" gracePeriod=30
Jul 28 15:05:51.062316 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062280    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"]
Jul 28 15:05:51.062695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062681    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49736725-8b0c-4686-8f48-22c1779f1a1c" containerName="maas-api"
Jul 28 15:05:51.062743 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062697    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49736725-8b0c-4686-8f48-22c1779f1a1c" containerName="maas-api"
Jul 28 15:05:51.062743 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062723    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" containerName="payload-processing"
Jul 28 15:05:51.062743 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062729    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" containerName="payload-processing"
Jul 28 15:05:51.062743 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062742    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b492af26-8598-46fb-b6bc-d22159cbb47a" containerName="payload-pre-processing"
Jul 28 15:05:51.062874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062749    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b492af26-8598-46fb-b6bc-d22159cbb47a" containerName="payload-pre-processing"
Jul 28 15:05:51.062874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062812    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b492af26-8598-46fb-b6bc-d22159cbb47a" containerName="payload-pre-processing"
Jul 28 15:05:51.062874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062819    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fffb50a-6c37-44e3-950c-ce2ceaa03dc3" containerName="payload-processing"
Jul 28 15:05:51.062874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.062826    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="49736725-8b0c-4686-8f48-22c1779f1a1c" containerName="maas-api"
Jul 28 15:05:51.065855 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.065839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:51.067909 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.067883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:05:51.073212 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.073183    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"]
Jul 28 15:05:51.152610 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.152567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k\" (UID: \"733f1354-dac1-4d56-abc0-c0dd708b4cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:51.253451 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.253416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k\" (UID: \"733f1354-dac1-4d56-abc0-c0dd708b4cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:51.254093 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.254072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k\" (UID: \"733f1354-dac1-4d56-abc0-c0dd708b4cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:51.378154 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.378073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:51.514671 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.514592    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"]
Jul 28 15:05:51.517194 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:05:51.517167    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733f1354_dac1_4d56_abc0_c0dd708b4cf2.slice/crio-ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68 WatchSource:0}: Error finding container ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68: Status 404 returned error can't find the container with id ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68
Jul 28 15:05:51.855075 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.855031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k" event={"ID":"733f1354-dac1-4d56-abc0-c0dd708b4cf2","Type":"ContainerStarted","Data":"39050f58cf3044cacf4b9a76c410c0c46035f84df8431144bc843d660586f60b"}
Jul 28 15:05:51.855460 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:51.855095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k" event={"ID":"733f1354-dac1-4d56-abc0-c0dd708b4cf2","Type":"ContainerStarted","Data":"ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68"}
Jul 28 15:05:52.862169 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:52.862136    2577 generic.go:356] "Generic (PLEG): container finished" podID="733f1354-dac1-4d56-abc0-c0dd708b4cf2" containerID="39050f58cf3044cacf4b9a76c410c0c46035f84df8431144bc843d660586f60b" exitCode=0
Jul 28 15:05:52.862632 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:52.862227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k" event={"ID":"733f1354-dac1-4d56-abc0-c0dd708b4cf2","Type":"ContainerDied","Data":"39050f58cf3044cacf4b9a76c410c0c46035f84df8431144bc843d660586f60b"}
Jul 28 15:05:54.001936 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.001902    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:54.080799 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.080763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca\") pod \"733f1354-dac1-4d56-abc0-c0dd708b4cf2\" (UID: \"733f1354-dac1-4d56-abc0-c0dd708b4cf2\") "
Jul 28 15:05:54.081147 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.081122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "733f1354-dac1-4d56-abc0-c0dd708b4cf2" (UID: "733f1354-dac1-4d56-abc0-c0dd708b4cf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:54.181496 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.181412    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f1354-dac1-4d56-abc0-c0dd708b4cf2-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:54.873365 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.873338    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"
Jul 28 15:05:54.873553 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.873329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k" event={"ID":"733f1354-dac1-4d56-abc0-c0dd708b4cf2","Type":"ContainerDied","Data":"ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68"}
Jul 28 15:05:54.873553 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:54.873460    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee24e03f620c7c8a4bd159232fd7a94d2e18f84faeb14c5ab78107615d3f4a68"
Jul 28 15:05:55.821106 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.821082    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:05:55.879996 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.879902    2577 generic.go:356] "Generic (PLEG): container finished" podID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerID="56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691" exitCode=0
Jul 28 15:05:55.879996 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.879959    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"
Jul 28 15:05:55.880247 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.879952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerDied","Data":"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"}
Jul 28 15:05:55.880247 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.880079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf" event={"ID":"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490","Type":"ContainerDied","Data":"a4fda60a59e751dfae6ba008179f50e62c8e21d475b382ccedc0a0ccca0965ca"}
Jul 28 15:05:55.880247 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.880111    2577 scope.go:117] "RemoveContainer" containerID="56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"
Jul 28 15:05:55.888764 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.888744    2577 scope.go:117] "RemoveContainer" containerID="7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"
Jul 28 15:05:55.896843 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.896821    2577 scope.go:117] "RemoveContainer" containerID="56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"
Jul 28 15:05:55.897036 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897161 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897161 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897161 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897161 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897161 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:05:55.897154    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691\": container with ID starting with 56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691 not found: ID does not exist" containerID="56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6fcg\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897183    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691"} err="failed to get container status \"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691\": rpc error: code = NotFound desc = could not find container \"56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691\": container with ID starting with 56f798eec0142ceab4e40fae68f936b046efc47fae2fb151dd7835e76b89c691 not found: ID does not exist"
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897207    2577 scope.go:117] "RemoveContainer" containerID="7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket\") pod \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\" (UID: \"2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490\") "
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897368    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:05:55.897447 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897417    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:55.897884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:05:55.897884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897738    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.897884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897758    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.897884 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.897840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:05:55.898159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.898044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:05:55.898159 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:05:55.898107    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79\": container with ID starting with 7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79 not found: ID does not exist" containerID="7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"
Jul 28 15:05:55.898250 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.898138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79"} err="failed to get container status \"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79\": rpc error: code = NotFound desc = could not find container \"7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79\": container with ID starting with 7ad57740dd859e6f6b8da0c24eca291e81c5f42ad50b6543e0ade80a19877e79 not found: ID does not exist"
Jul 28 15:05:55.899463 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.899437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:05:55.899867 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.899847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:05:55.900045 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.900025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg" (OuterVolumeSpecName: "kube-api-access-g6fcg") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "kube-api-access-g6fcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:05:55.900147 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.900015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" (UID: "2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:05:55.999269 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999232    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999269 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999261    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999269 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999269    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999269 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999280    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fcg\" (UniqueName: \"kubernetes.io/projected/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-kube-api-access-g6fcg\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999597 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999289    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999597 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999297    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:55.999597 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:55.999305    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:05:56.204206 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:56.204173    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:05:56.209522 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:56.209494    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7d80d5-openshift-default-555c8769b8-z5bdf"]
Jul 28 15:05:56.791686 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:05:56.791653    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" path="/var/lib/kubelet/pods/2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490/volumes"
Jul 28 15:06:06.612769 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.612686    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:06:06.613257 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.612960    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" podUID="e4c8cd5a-8e62-4685-886f-444107861fc5" containerName="payload-pre-processing" containerID="cri-o://4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711" gracePeriod=30
Jul 28 15:06:06.729786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.729759    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"]
Jul 28 15:06:06.733688 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.733660    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-84fa8d-5jr7k"]
Jul 28 15:06:06.788786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.788352    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733f1354-dac1-4d56-abc0-c0dd708b4cf2" path="/var/lib/kubelet/pods/733f1354-dac1-4d56-abc0-c0dd708b4cf2/volumes"
Jul 28 15:06:06.880702 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.880672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:06:06.934695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.934666    2577 generic.go:356] "Generic (PLEG): container finished" podID="e4c8cd5a-8e62-4685-886f-444107861fc5" containerID="4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711" exitCode=0
Jul 28 15:06:06.934865 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.934704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" event={"ID":"e4c8cd5a-8e62-4685-886f-444107861fc5","Type":"ContainerDied","Data":"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"}
Jul 28 15:06:06.934865 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.934710    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"
Jul 28 15:06:06.934865 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.934746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82" event={"ID":"e4c8cd5a-8e62-4685-886f-444107861fc5","Type":"ContainerDied","Data":"4ab7385cbd9da2caed1a329a3dd4b67d078ae84ab48aa40e1bc6bb7ce780f072"}
Jul 28 15:06:06.934865 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.934768    2577 scope.go:117] "RemoveContainer" containerID="4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"
Jul 28 15:06:06.942960 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.942937    2577 scope.go:117] "RemoveContainer" containerID="4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"
Jul 28 15:06:06.943224 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:06:06.943206    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711\": container with ID starting with 4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711 not found: ID does not exist" containerID="4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"
Jul 28 15:06:06.943296 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:06.943231    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711"} err="failed to get container status \"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711\": rpc error: code = NotFound desc = could not find container \"4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711\": container with ID starting with 4b08f54594cdcb5f97f0e397f94f674b2f81b594d81223ee8e45b44c09d3a711 not found: ID does not exist"
Jul 28 15:06:07.000310 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.000278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz\") pod \"e4c8cd5a-8e62-4685-886f-444107861fc5\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") "
Jul 28 15:06:07.000476 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.000329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume\") pod \"e4c8cd5a-8e62-4685-886f-444107861fc5\" (UID: \"e4c8cd5a-8e62-4685-886f-444107861fc5\") "
Jul 28 15:06:07.000766 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.000738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4c8cd5a-8e62-4685-886f-444107861fc5" (UID: "e4c8cd5a-8e62-4685-886f-444107861fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:06:07.002398 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.002374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz" (OuterVolumeSpecName: "kube-api-access-hpftz") pod "e4c8cd5a-8e62-4685-886f-444107861fc5" (UID: "e4c8cd5a-8e62-4685-886f-444107861fc5"). InnerVolumeSpecName "kube-api-access-hpftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:06:07.101611 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.101570    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpftz\" (UniqueName: \"kubernetes.io/projected/e4c8cd5a-8e62-4685-886f-444107861fc5-kube-api-access-hpftz\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:07.101611 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.101604    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c8cd5a-8e62-4685-886f-444107861fc5-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:07.255324 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.255287    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:06:07.256866 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:07.256843    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-84fa8d-5665ccf796-nnf82"]
Jul 28 15:06:08.785143 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:08.785108    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c8cd5a-8e62-4685-886f-444107861fc5" path="/var/lib/kubelet/pods/e4c8cd5a-8e62-4685-886f-444107861fc5/volumes"
Jul 28 15:06:13.385043 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:13.385008    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:06:13.385424 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:13.385236    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy" containerID="cri-o://71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472" gracePeriod=30
Jul 28 15:06:14.360927 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.360887    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:06:14.361460 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361443    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733f1354-dac1-4d56-abc0-c0dd708b4cf2" containerName="revoke-keys"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="733f1354-dac1-4d56-abc0-c0dd708b4cf2" containerName="revoke-keys"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361488    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c8cd5a-8e62-4685-886f-444107861fc5" containerName="payload-pre-processing"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361497    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c8cd5a-8e62-4685-886f-444107861fc5" containerName="payload-pre-processing"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361521    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:14.361568 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361558    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:14.361845 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:14.361845 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361646    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c8cd5a-8e62-4685-886f-444107861fc5" containerName="payload-pre-processing"
Jul 28 15:06:14.361845 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.361657    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="733f1354-dac1-4d56-abc0-c0dd708b4cf2" containerName="revoke-keys"
Jul 28 15:06:14.366211 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.366188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.368455 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.368429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fd42ef-openshift-default-dockercfg-wrsvh\""
Jul 28 15:06:14.374551 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.374509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:06:14.467653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.467653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.468129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.467872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nlj5\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569223 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569415 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nlj5\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569715 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569715 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569715 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569715 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569715 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569909 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.569909 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.569838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.570219 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.570184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.571822 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.571802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.572186 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.572159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.587464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.587431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nlj5\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.587464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.587458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token\") pod \"e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") " pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.679074 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.678978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:14.813635 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.813609    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:06:14.816239 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:14.816204    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cdc7b5_db51_49e2_8907_487d448723dd.slice/crio-5598eba56b696d7e677884e3065c620c0d07d27be304167cc3663a33e86479fc WatchSource:0}: Error finding container 5598eba56b696d7e677884e3065c620c0d07d27be304167cc3663a33e86479fc: Status 404 returned error can't find the container with id 5598eba56b696d7e677884e3065c620c0d07d27be304167cc3663a33e86479fc
Jul 28 15:06:14.818426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.818393    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:14.818492 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.818473    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:14.818553 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.818503    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:14.978940 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.978892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerStarted","Data":"1529ddb503368923c926fed760a55139b0514b1e61d51d7870f2ebe0253147ae"}
Jul 28 15:06:14.979144 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:14.978958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerStarted","Data":"5598eba56b696d7e677884e3065c620c0d07d27be304167cc3663a33e86479fc"}
Jul 28 15:06:15.000956 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:15.000889    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podStartSLOduration=1.000863709 podStartE2EDuration="1.000863709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:14.997599594 +0000 UTC m=+2612.683677708" watchObservedRunningTime="2026-07-28 15:06:15.000863709 +0000 UTC m=+2612.686941818"
Jul 28 15:06:15.679795 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:15.679750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:15.685134 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:15.685107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:15.983695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:15.983660    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:15.984746 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:15.984723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:16.586114 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:16.586076    2577 patch_prober.go:28] interesting pod/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 15:06:16.586288 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:16.586132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:06:17.102813 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.102777    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:06:17.103459 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.103436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dec8dbb-3dd5-4a1f-9679-11f8fbf0e490" containerName="istio-proxy"
Jul 28 15:06:17.106841 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.106822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.109291 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.109272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fd42ef-serving-cert\""
Jul 28 15:06:17.109400 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.109390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk5gs\""
Jul 28 15:06:17.116371 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.116336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:06:17.195259 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.195215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.195462 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.195270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdr9\" (UniqueName: \"kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.195462 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.195298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.296705 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.296667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.296892 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.296716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdr9\" (UniqueName: \"kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.296892 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.296735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.297338 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.297315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.299254 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.299227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.308895 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.308869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdr9\" (UniqueName: \"kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9\") pod \"maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.419564 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.419436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:17.563720 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.563682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:06:17.567640 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:17.567597    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb78b86_740c_46d8_89b3_1f0a6eb5f88f.slice/crio-64047eca30fe91c926c765737f322f59242a5540fb253e6e71b45227771a27f3 WatchSource:0}: Error finding container 64047eca30fe91c926c765737f322f59242a5540fb253e6e71b45227771a27f3: Status 404 returned error can't find the container with id 64047eca30fe91c926c765737f322f59242a5540fb253e6e71b45227771a27f3
Jul 28 15:06:17.792464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.792429    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:06:17.797016 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.796999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:17.799924 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.799894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-fd42ef-dockercfg-hddb7\""
Jul 28 15:06:17.800075 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.800064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-fd42ef\""
Jul 28 15:06:17.804961 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.804933    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:06:17.811336 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.811311    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:06:17.821614 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.821579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:17.824859 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.824831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:06:17.903825 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.903784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:17.903825 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.903828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:17.904105 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.903862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2rs\" (UniqueName: \"kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:17.904105 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.903895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc6j\" (UniqueName: \"kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:17.995238 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.995207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" event={"ID":"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f","Type":"ContainerStarted","Data":"2037f539d5b00afcdd7686d2d3f3a7c3af4bd24964a5663534d925286de777fd"}
Jul 28 15:06:17.995410 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.995257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" event={"ID":"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f","Type":"ContainerStarted","Data":"64047eca30fe91c926c765737f322f59242a5540fb253e6e71b45227771a27f3"}
Jul 28 15:06:17.995601 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:17.995579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:18.004563 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.004520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:18.004691 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.004571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:18.004691 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.004643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2rs\" (UniqueName: \"kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:18.004816 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.004795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc6j\" (UniqueName: \"kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:18.005265 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.005242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:18.005359 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.005328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:18.012634 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.012588    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" podStartSLOduration=0.656577426 podStartE2EDuration="1.012571568s" totalImagesPullingTime="355.994142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:18.010786902 +0000 UTC m=+2615.696865011" watchObservedRunningTime="2026-07-28 15:06:18.012571568 +0000 UTC m=+2615.698649681"
Jul 28 15:06:18.015829 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.015760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2rs\" (UniqueName: \"kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs\") pod \"payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:18.015983 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.015834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc6j\" (UniqueName: \"kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j\") pod \"payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") " pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:18.109108 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.109027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:18.133519 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.133457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:18.262613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.262576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:06:18.265419 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:18.265384    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96298cd_c845_41c3_9fce_d1b06e7b56fd.slice/crio-d9ab65c19a379730ada6fc38e59ff0799218faddf02c5fb207d5cc1d2e93976e WatchSource:0}: Error finding container d9ab65c19a379730ada6fc38e59ff0799218faddf02c5fb207d5cc1d2e93976e: Status 404 returned error can't find the container with id d9ab65c19a379730ada6fc38e59ff0799218faddf02c5fb207d5cc1d2e93976e
Jul 28 15:06:18.296546 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.296509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:06:18.299766 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:18.299737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33948fa9_dd3e_4495_b5c4_d918705fe3ce.slice/crio-4e766ecae21b49ba24fbc12c800874203c091a036326528b97e38bbcc32f8667 WatchSource:0}: Error finding container 4e766ecae21b49ba24fbc12c800874203c091a036326528b97e38bbcc32f8667: Status 404 returned error can't find the container with id 4e766ecae21b49ba24fbc12c800874203c091a036326528b97e38bbcc32f8667
Jul 28 15:06:18.655158 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.655126    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:06:18.711794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.711754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.711794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.711792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712049 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.711841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712049 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.711860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712049 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.711971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82dp4\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712049 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:06:18.712252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs\") pod \"a6163563-0de0-46b6-aa18-95773f8f9854\" (UID: \"a6163563-0de0-46b6-aa18-95773f8f9854\") "
Jul 28 15:06:18.712252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:06:18.712495 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:06:18.712670 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712653    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6163563-0de0-46b6-aa18-95773f8f9854-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.712746 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712676    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.712746 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712691    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.712746 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.712722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:06:18.713312 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.713219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:06:18.714567 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.714521    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4" (OuterVolumeSpecName: "kube-api-access-82dp4") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "kube-api-access-82dp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:06:18.714772 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.714751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:06:18.714772 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.714753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:06:18.714907 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.714858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6163563-0de0-46b6-aa18-95773f8f9854" (UID: "a6163563-0de0-46b6-aa18-95773f8f9854"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:06:18.814298 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814262    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.814298 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814298    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.814502 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814311    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.814502 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814326    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6163563-0de0-46b6-aa18-95773f8f9854-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.814502 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814338    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6163563-0de0-46b6-aa18-95773f8f9854-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:18.814502 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:18.814352    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82dp4\" (UniqueName: \"kubernetes.io/projected/a6163563-0de0-46b6-aa18-95773f8f9854-kube-api-access-82dp4\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:06:19.002420 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.002379    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6163563-0de0-46b6-aa18-95773f8f9854" containerID="71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472" exitCode=0
Jul 28 15:06:19.002654 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.002425    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"
Jul 28 15:06:19.002654 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.002467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerDied","Data":"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"}
Jul 28 15:06:19.002654 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.002557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz" event={"ID":"a6163563-0de0-46b6-aa18-95773f8f9854","Type":"ContainerDied","Data":"97de50031da3b82da1d9c3207ea88ca35a21c0090660ef1a226458db4359bead"}
Jul 28 15:06:19.002654 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.002595    2577 scope.go:117] "RemoveContainer" containerID="71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"
Jul 28 15:06:19.004479 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.004451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" event={"ID":"33948fa9-dd3e-4495-b5c4-d918705fe3ce","Type":"ContainerStarted","Data":"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"}
Jul 28 15:06:19.004624 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.004510    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:19.004624 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.004563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" event={"ID":"33948fa9-dd3e-4495-b5c4-d918705fe3ce","Type":"ContainerStarted","Data":"4e766ecae21b49ba24fbc12c800874203c091a036326528b97e38bbcc32f8667"}
Jul 28 15:06:19.006349 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.006323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" event={"ID":"b96298cd-c845-41c3-9fce-d1b06e7b56fd","Type":"ContainerStarted","Data":"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"}
Jul 28 15:06:19.006459 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.006377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" event={"ID":"b96298cd-c845-41c3-9fce-d1b06e7b56fd","Type":"ContainerStarted","Data":"d9ab65c19a379730ada6fc38e59ff0799218faddf02c5fb207d5cc1d2e93976e"}
Jul 28 15:06:19.006459 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.006410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:19.012615 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.012596    2577 scope.go:117] "RemoveContainer" containerID="9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"
Jul 28 15:06:19.018031 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.017996    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:06:19.021009 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.020982    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-84fa8d-openshift-default-659c9b64f6-cwwhz"]
Jul 28 15:06:19.022780 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.022755    2577 scope.go:117] "RemoveContainer" containerID="71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"
Jul 28 15:06:19.023114 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:06:19.023087    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472\": container with ID starting with 71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472 not found: ID does not exist" containerID="71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"
Jul 28 15:06:19.023178 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.023115    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472"} err="failed to get container status \"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472\": rpc error: code = NotFound desc = could not find container \"71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472\": container with ID starting with 71124b45f1a5d34ecb9912b602aeab725a06216946ff46571646069338554472 not found: ID does not exist"
Jul 28 15:06:19.023178 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.023134    2577 scope.go:117] "RemoveContainer" containerID="9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"
Jul 28 15:06:19.023392 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:06:19.023374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de\": container with ID starting with 9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de not found: ID does not exist" containerID="9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"
Jul 28 15:06:19.023450 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.023399    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de"} err="failed to get container status \"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de\": rpc error: code = NotFound desc = could not find container \"9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de\": container with ID starting with 9d3fabfffbcf3c7430d176f7ed06fc848dfe6531392ebbebb2b0fff1446236de not found: ID does not exist"
Jul 28 15:06:19.032966 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.032908    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" podStartSLOduration=2.032889176 podStartE2EDuration="2.032889176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:19.032138417 +0000 UTC m=+2616.718216531" watchObservedRunningTime="2026-07-28 15:06:19.032889176 +0000 UTC m=+2616.718967289"
Jul 28 15:06:19.046030 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:19.045977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" podStartSLOduration=2.045959176 podStartE2EDuration="2.045959176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:19.045447251 +0000 UTC m=+2616.731525389" watchObservedRunningTime="2026-07-28 15:06:19.045959176 +0000 UTC m=+2616.732037290"
Jul 28 15:06:20.785430 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:20.785400    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" path="/var/lib/kubelet/pods/a6163563-0de0-46b6-aa18-95773f8f9854/volumes"
Jul 28 15:06:24.018924 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:24.018881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:06:30.014955 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:30.014923    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:06:30.015344 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:30.014994    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:06:30.985737 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:30.985697    2577 patch_prober.go:28] interesting pod/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Jul 28 15:06:30.985975 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:30.985772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Jul 28 15:06:31.073674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.073642    2577 generic.go:356] "Generic (PLEG): container finished" podID="03cdc7b5-db51-49e2-8907-487d448723dd" containerID="1529ddb503368923c926fed760a55139b0514b1e61d51d7870f2ebe0253147ae" exitCode=0
Jul 28 15:06:31.074066 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.073718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerDied","Data":"1529ddb503368923c926fed760a55139b0514b1e61d51d7870f2ebe0253147ae"}
Jul 28 15:06:31.074127 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.074113    2577 scope.go:117] "RemoveContainer" containerID="1529ddb503368923c926fed760a55139b0514b1e61d51d7870f2ebe0253147ae"
Jul 28 15:06:31.075149 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.075120    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:31.075219 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.075201    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:31.075258 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:31.075242    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:32.080424 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:32.080391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerStarted","Data":"5c4684609cf21fc12c167fb14c76be9ec02324c0fbffc2bb226ab7441c7593b4"}
Jul 28 15:06:32.679687 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:32.679648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:32.679870 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:32.679805    2577 patch_prober.go:28] interesting pod/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Jul 28 15:06:32.679870 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:32.679860    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Jul 28 15:06:33.680124 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:33.680089    2577 patch_prober.go:28] interesting pod/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Jul 28 15:06:33.680642 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:33.680148    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Jul 28 15:06:34.680005 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:34.679968    2577 patch_prober.go:28] interesting pod/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Jul 28 15:06:34.680186 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:34.680024    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Jul 28 15:06:35.680320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:35.680284    2577 patch_prober.go:28] interesting pod/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Jul 28 15:06:35.680835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:35.680347    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Jul 28 15:06:36.683814 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:36.683784    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:36.684205 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:36.684055    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:36.684740 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:36.684723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:06:42.939554 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.939498    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:06:42.940114 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940084    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.940114 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940110    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.940320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940125    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.940320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940132    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.940320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940229    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.940320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.940241    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6163563-0de0-46b6-aa18-95773f8f9854" containerName="istio-proxy"
Jul 28 15:06:42.942475 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.942458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:42.944644 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.944614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bc713f-openshift-default-dockercfg-92t9n\""
Jul 28 15:06:42.954823 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:42.954796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:06:43.044427 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044951 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.044951 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.044819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5tq\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.146689 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.146649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.147017 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.146996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5tq\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.147192 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.147177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.147422 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.147349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.147658 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.147529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148170 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148577 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148665 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148729 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148840 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.148908 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.148843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.149320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.149291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.150902 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.150875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.151348 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.151325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.155341 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.155317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.155626 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.155603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5tq\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq\") pod \"e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") " pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.257675 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.257636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:43.399872 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.399847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:06:43.402335 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:43.402303    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7acabd4_df78_4a83_8d5a_1d379abd5205.slice/crio-e1bd9a2b4a1402bdee67451ac7312504a0a2cc13c55fcb689d1f7ccf60ac819c WatchSource:0}: Error finding container e1bd9a2b4a1402bdee67451ac7312504a0a2cc13c55fcb689d1f7ccf60ac819c: Status 404 returned error can't find the container with id e1bd9a2b4a1402bdee67451ac7312504a0a2cc13c55fcb689d1f7ccf60ac819c
Jul 28 15:06:43.404703 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.404666    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:43.404799 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.404740    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:43.404799 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:43.404770    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:06:44.156106 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:44.156066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerStarted","Data":"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"}
Jul 28 15:06:44.156588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:44.156124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerStarted","Data":"e1bd9a2b4a1402bdee67451ac7312504a0a2cc13c55fcb689d1f7ccf60ac819c"}
Jul 28 15:06:44.174278 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:44.174200    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podStartSLOduration=2.174185525 podStartE2EDuration="2.174185525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:44.172091797 +0000 UTC m=+2641.858169910" watchObservedRunningTime="2026-07-28 15:06:44.174185525 +0000 UTC m=+2641.860263638"
Jul 28 15:06:44.257904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:44.257867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:44.263194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:44.263165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:45.160733 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:45.160699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:45.161722 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:45.161705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:06:46.993868 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:46.993836    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:06:47.002286 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.002235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.005035 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.004991    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-bc713f\""
Jul 28 15:06:47.005191 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.005017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-bc713f-dockercfg-t4z8g\""
Jul 28 15:06:47.009258 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.009231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:06:47.185287 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.185252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.185499 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.185372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr5b\" (UniqueName: \"kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.286691 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.286605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr5b\" (UniqueName: \"kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.286691 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.286675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.287349 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.287329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.293962 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.293931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr5b\" (UniqueName: \"kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b\") pod \"payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.315854 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.315815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:47.464951 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:47.464917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:06:47.466766 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:06:47.466724    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3517b2_6d18_4a50_a1aa_dab3534da925.slice/crio-d42b1e7119133bbec589c2944c85445b1b3120c5ec7cb0557d9da03dad9ec79b WatchSource:0}: Error finding container d42b1e7119133bbec589c2944c85445b1b3120c5ec7cb0557d9da03dad9ec79b: Status 404 returned error can't find the container with id d42b1e7119133bbec589c2944c85445b1b3120c5ec7cb0557d9da03dad9ec79b
Jul 28 15:06:48.176910 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:48.176868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" event={"ID":"eb3517b2-6d18-4a50-a1aa-dab3534da925","Type":"ContainerStarted","Data":"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"}
Jul 28 15:06:48.177335 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:48.176932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" event={"ID":"eb3517b2-6d18-4a50-a1aa-dab3534da925","Type":"ContainerStarted","Data":"d42b1e7119133bbec589c2944c85445b1b3120c5ec7cb0557d9da03dad9ec79b"}
Jul 28 15:06:48.177335 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:48.176975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:06:48.196256 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:48.196204    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" podStartSLOduration=2.19618549 podStartE2EDuration="2.19618549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:06:48.194255161 +0000 UTC m=+2645.880333275" watchObservedRunningTime="2026-07-28 15:06:48.19618549 +0000 UTC m=+2645.882263675"
Jul 28 15:06:59.184673 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:06:59.184638    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:07:01.111852 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.111815    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:43920->10.133.0.82:15021: read: connection reset by peer" start-of-body=
Jul 28 15:07:01.112252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.111900    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:43920->10.133.0.82:15021: read: connection reset by peer"
Jul 28 15:07:01.254227 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.254195    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerID="8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4" exitCode=0
Jul 28 15:07:01.254405 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.254269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerDied","Data":"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"}
Jul 28 15:07:01.254711 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.254697    2577 scope.go:117] "RemoveContainer" containerID="8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"
Jul 28 15:07:01.255690 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.255668    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:07:01.255747 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.255730    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:07:01.255790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:01.255763    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:07:02.260823 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:02.260787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerStarted","Data":"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"}
Jul 28 15:07:03.258332 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:03.258284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:07:03.258623 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:03.258598    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 28 15:07:03.258727 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:03.258660    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 28 15:07:04.258376 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:04.258339    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 28 15:07:04.258829 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:04.258395    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 28 15:07:05.258972 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:05.258934    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 28 15:07:05.259370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:05.258994    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 28 15:07:06.261746 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:06.261712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:07:06.262252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:06.262123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:07:06.262846 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:06.262827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:07:42.931380 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:42.931300    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:07:42.935048 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:42.935023    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:07:42.939812 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:42.939771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:07:42.942912 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:07:42.942893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:10:43.209031 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:10:43.208944    2577 scope.go:117] "RemoveContainer" containerID="bb4c40d0c706f19890b3fed631bdb3387fc968aa7a90a32e83da3aca98c85c33"
Jul 28 15:12:42.962146 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:12:42.962119    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:12:42.965763 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:12:42.965745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:12:42.971562 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:12:42.971524    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:12:42.974704 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:12:42.974687    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:12:43.238494 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:12:43.238408    2577 scope.go:117] "RemoveContainer" containerID="39050f58cf3044cacf4b9a76c410c0c46035f84df8431144bc843d660586f60b"
Jul 28 15:15:00.131345 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.131311    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"]
Jul 28 15:15:00.135176 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.135157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"
Jul 28 15:15:00.137321 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.137304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:15:00.141734 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.141643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"]
Jul 28 15:15:00.147406 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.147384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"
Jul 28 15:15:00.286116 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.286083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"]
Jul 28 15:15:00.288500 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:15:00.288471    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f3e510_e0f4_457c_8445_5868db7f7b42.slice/crio-ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd WatchSource:0}: Error finding container ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd: Status 404 returned error can't find the container with id ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd
Jul 28 15:15:00.579163 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:00.579128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7" event={"ID":"b3f3e510-e0f4-457c-8445-5868db7f7b42","Type":"ContainerStarted","Data":"ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd"}
Jul 28 15:15:01.586739 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:01.586709    2577 generic.go:356] "Generic (PLEG): container finished" podID="b3f3e510-e0f4-457c-8445-5868db7f7b42" containerID="ba5deed91e8d5f80db14907003915da48434fcd3d6080081db5534063facf65d" exitCode=0
Jul 28 15:15:01.587130 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:01.586802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7" event={"ID":"b3f3e510-e0f4-457c-8445-5868db7f7b42","Type":"ContainerDied","Data":"ba5deed91e8d5f80db14907003915da48434fcd3d6080081db5534063facf65d"}
Jul 28 15:15:02.737901 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:02.737856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"
Jul 28 15:15:03.597948 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:03.597919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7"
Jul 28 15:15:03.597948 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:03.597931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vwfp7" event={"ID":"b3f3e510-e0f4-457c-8445-5868db7f7b42","Type":"ContainerDied","Data":"ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd"}
Jul 28 15:15:03.598228 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:15:03.597976    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab39d61418452a90cc0a17afa132b1b5ccf75d0c262831b9528aca72c35bafbd"
Jul 28 15:16:43.271111 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.271028    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:16:43.271653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.271636    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f3e510-e0f4-457c-8445-5868db7f7b42" containerName="cleanup"
Jul 28 15:16:43.271706 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.271656    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f3e510-e0f4-457c-8445-5868db7f7b42" containerName="cleanup"
Jul 28 15:16:43.271771 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.271761    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f3e510-e0f4-457c-8445-5868db7f7b42" containerName="cleanup"
Jul 28 15:16:43.275144 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.275124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.277373 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.277346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-968c04-serving-cert\""
Jul 28 15:16:43.290725 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.290690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:16:43.379698 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.379663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64v4h\" (UniqueName: \"kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.379874 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.379713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.379934 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.379908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.480888 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.480858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64v4h\" (UniqueName: \"kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.481062 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.480900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.481062 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.480990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.481144 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:16:43.481081    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-serving-cert: secret "maas-api-e2e-ipp-a-968c04-serving-cert" not found
Jul 28 15:16:43.481189 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:16:43.481147    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls podName:e339bde1-af9e-4022-9358-618ae503ae63 nodeName:}" failed. No retries permitted until 2026-07-28 15:16:43.981131961 +0000 UTC m=+3241.667210052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls") pod "maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" (UID: "e339bde1-af9e-4022-9358-618ae503ae63") : secret "maas-api-e2e-ipp-a-968c04-serving-cert" not found
Jul 28 15:16:43.481611 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.481591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.489488 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.489459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64v4h\" (UniqueName: \"kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.985190 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.985154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:43.987658 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:43.987632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") pod \"maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:44.028157 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.028123    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:16:44.031623 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.031605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.034382 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.034362    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-968c04\""
Jul 28 15:16:44.034488 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.034438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-968c04-dockercfg-6qc9q\""
Jul 28 15:16:44.039346 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.039323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:16:44.071060 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.071024    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:16:44.074517 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.074495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.082713 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.082690    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:16:44.186822 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.186787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9q4m\" (UniqueName: \"kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.187002 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.186842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.187002 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.186939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.187090 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.187026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd77t\" (UniqueName: \"kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.190662 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.190635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:44.288279 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.288244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd77t\" (UniqueName: \"kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.288872 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.288289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9q4m\" (UniqueName: \"kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.288872 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.288406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.288872 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.288478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.289409 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.289383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.289500 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.289390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.296372 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.296335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9q4m\" (UniqueName: \"kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m\") pod \"payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.296823 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.296800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd77t\" (UniqueName: \"kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t\") pod \"payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.329154 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.329114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:16:44.331039 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:16:44.331014    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode339bde1_af9e_4022_9358_618ae503ae63.slice/crio-75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75 WatchSource:0}: Error finding container 75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75: Status 404 returned error can't find the container with id 75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75
Jul 28 15:16:44.332448 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.332432    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:16:44.344283 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.344258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:44.386523 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.386487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:44.696133 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.696109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:16:44.697988 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:16:44.697965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d438cb_6502_4598_b4d5_b3e2f60b019f.slice/crio-ef0e3074ed11fe2b1f8d89604b8442fa3f97d74ef98934ad464c493502a2d4e2 WatchSource:0}: Error finding container ef0e3074ed11fe2b1f8d89604b8442fa3f97d74ef98934ad464c493502a2d4e2: Status 404 returned error can't find the container with id ef0e3074ed11fe2b1f8d89604b8442fa3f97d74ef98934ad464c493502a2d4e2
Jul 28 15:16:44.741469 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:44.741443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:16:44.744373 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:16:44.744346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6cecae_6f51_4ab1_840d_df68714dabf8.slice/crio-ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9 WatchSource:0}: Error finding container ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9: Status 404 returned error can't find the container with id ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9
Jul 28 15:16:45.082011 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.081974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" event={"ID":"69d438cb-6502-4598-b4d5-b3e2f60b019f","Type":"ContainerStarted","Data":"acf7823eb38f0e9decbcb3820c807b4b7443c48ad225f414bbfc039a1ffe46df"}
Jul 28 15:16:45.082213 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.082030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" event={"ID":"69d438cb-6502-4598-b4d5-b3e2f60b019f","Type":"ContainerStarted","Data":"ef0e3074ed11fe2b1f8d89604b8442fa3f97d74ef98934ad464c493502a2d4e2"}
Jul 28 15:16:45.082213 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.082094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:16:45.083313 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.083288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" event={"ID":"e339bde1-af9e-4022-9358-618ae503ae63","Type":"ContainerStarted","Data":"75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75"}
Jul 28 15:16:45.084928 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.084902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" event={"ID":"bc6cecae-6f51-4ab1-840d-df68714dabf8","Type":"ContainerStarted","Data":"5ac1d42751484603a7e5be6593dc3e0faae63e6f5654d47429fd337395b363cc"}
Jul 28 15:16:45.085047 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.084956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" event={"ID":"bc6cecae-6f51-4ab1-840d-df68714dabf8","Type":"ContainerStarted","Data":"ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9"}
Jul 28 15:16:45.085047 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.084993    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:16:45.098206 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.098164    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" podStartSLOduration=1.098148794 podStartE2EDuration="1.098148794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:16:45.095733231 +0000 UTC m=+3242.781811355" watchObservedRunningTime="2026-07-28 15:16:45.098148794 +0000 UTC m=+3242.784226904"
Jul 28 15:16:45.110216 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:45.110148    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" podStartSLOduration=1.110129483 podStartE2EDuration="1.110129483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:16:45.109128102 +0000 UTC m=+3242.795206216" watchObservedRunningTime="2026-07-28 15:16:45.110129483 +0000 UTC m=+3242.796207587"
Jul 28 15:16:46.092177 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:46.092130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" event={"ID":"e339bde1-af9e-4022-9358-618ae503ae63","Type":"ContainerStarted","Data":"03c83137519415b0179505fb6f1d7378a148791e0a83892aefcd7358ad4a3ea5"}
Jul 28 15:16:46.092731 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:46.092214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:46.110129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:46.110073    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" podStartSLOduration=2.179606128 podStartE2EDuration="3.110057036s" totalImagesPullingTime="930.450908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:16:46.108342889 +0000 UTC m=+3243.794421002" watchObservedRunningTime="2026-07-28 15:16:46.110057036 +0000 UTC m=+3243.796135148"
Jul 28 15:16:52.102690 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:52.102661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:16:56.094265 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:56.094219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.86:9004: connect: connection refused"
Jul 28 15:16:56.094801 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:16:56.094656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:17:06.093784 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:06.093752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:17:42.994675 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:42.994643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:17:42.998303 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:42.998283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:17:43.004399 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:43.004373    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:17:43.008075 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:43.008050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:17:44.537425 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:17:44.537390    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:18:41.999127 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:41.999094    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq_bc6cecae-6f51-4ab1-840d-df68714dabf8/payload-processing/0.log"
Jul 28 15:18:42.122997 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:42.122965    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:18:45.719237 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.719197    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:18:45.723579 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.723560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.725826 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.725799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-adb651-serving-cert\""
Jul 28 15:18:45.731315 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.731290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:18:45.836386 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.836356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.836596 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.836411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm6w\" (UniqueName: \"kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.836596 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.836572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.938013 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.937984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.938013 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.938026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.938281 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.938088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm6w\" (UniqueName: \"kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.938765 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.938742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.940959 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.940933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:45.945441 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:45.945417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm6w\" (UniqueName: \"kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w\") pod \"maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:46.038345 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.038254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:46.179153 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.179125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:18:46.181212 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:18:46.181181    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583769d7_6761_4ca1_9181_1ea37a65ca0a.slice/crio-727eb638f84566c696a09fbbe169b99cc9cc5dc3279c9cbb644bc4778126e50c WatchSource:0}: Error finding container 727eb638f84566c696a09fbbe169b99cc9cc5dc3279c9cbb644bc4778126e50c: Status 404 returned error can't find the container with id 727eb638f84566c696a09fbbe169b99cc9cc5dc3279c9cbb644bc4778126e50c
Jul 28 15:18:46.413457 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.413371    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:18:46.418140 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.418119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.420293 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.420265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-adb651\""
Jul 28 15:18:46.420428 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.420354    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-adb651-dockercfg-7w72h\""
Jul 28 15:18:46.426342 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.426312    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:18:46.445183 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.445153    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:18:46.448854 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.448833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.458971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.458949    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:18:46.543023 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.542989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.543230 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.543040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrh4\" (UniqueName: \"kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.543230 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.543141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcskg\" (UniqueName: \"kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.543230 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.543166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.644670 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.644642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrh4\" (UniqueName: \"kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.644786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.644735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcskg\" (UniqueName: \"kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.644786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.644760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.644868 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.644853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.645468 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.645450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.645514 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.645496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.652809 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.652786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcskg\" (UniqueName: \"kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg\") pod \"payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.653006 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.652986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrh4\" (UniqueName: \"kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4\") pod \"payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.702999 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.702902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" event={"ID":"583769d7-6761-4ca1-9181-1ea37a65ca0a","Type":"ContainerStarted","Data":"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"}
Jul 28 15:18:46.702999 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.702961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" event={"ID":"583769d7-6761-4ca1-9181-1ea37a65ca0a","Type":"ContainerStarted","Data":"727eb638f84566c696a09fbbe169b99cc9cc5dc3279c9cbb644bc4778126e50c"}
Jul 28 15:18:46.702999 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.702984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:46.719607 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.719521    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" podStartSLOduration=1.307626703 podStartE2EDuration="1.71949833s" totalImagesPullingTime="411.871627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:18:46.718244987 +0000 UTC m=+3364.404323101" watchObservedRunningTime="2026-07-28 15:18:46.71949833 +0000 UTC m=+3364.405576445"
Jul 28 15:18:46.732544 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.732502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:46.761679 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.761649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:46.892517 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.892484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:18:46.895138 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:18:46.895107    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6af3fc_8a0b_450b_9e3d_c27fe7722d93.slice/crio-556f912f7d2dfbf4fc7fb4b59e0b81a0ebc7ce785c4567c802af7349c18e1d99 WatchSource:0}: Error finding container 556f912f7d2dfbf4fc7fb4b59e0b81a0ebc7ce785c4567c802af7349c18e1d99: Status 404 returned error can't find the container with id 556f912f7d2dfbf4fc7fb4b59e0b81a0ebc7ce785c4567c802af7349c18e1d99
Jul 28 15:18:46.926023 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:46.925997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:18:46.927718 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:18:46.927689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cc415d_b2df_410e_b82d_f69b90b7bf39.slice/crio-ea875772cdaa379df0a021c95f00f4da533c5bd0874f3371238349645f2f1e8d WatchSource:0}: Error finding container ea875772cdaa379df0a021c95f00f4da533c5bd0874f3371238349645f2f1e8d: Status 404 returned error can't find the container with id ea875772cdaa379df0a021c95f00f4da533c5bd0874f3371238349645f2f1e8d
Jul 28 15:18:47.710153 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.710115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" event={"ID":"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93","Type":"ContainerStarted","Data":"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"}
Jul 28 15:18:47.710467 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.710178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" event={"ID":"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93","Type":"ContainerStarted","Data":"556f912f7d2dfbf4fc7fb4b59e0b81a0ebc7ce785c4567c802af7349c18e1d99"}
Jul 28 15:18:47.710467 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.710208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:18:47.711921 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.711891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" event={"ID":"26cc415d-b2df-410e-b82d-f69b90b7bf39","Type":"ContainerStarted","Data":"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"}
Jul 28 15:18:47.712072 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.711943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" event={"ID":"26cc415d-b2df-410e-b82d-f69b90b7bf39","Type":"ContainerStarted","Data":"ea875772cdaa379df0a021c95f00f4da533c5bd0874f3371238349645f2f1e8d"}
Jul 28 15:18:47.712136 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.712086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:47.725228 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.725184    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" podStartSLOduration=1.725168706 podStartE2EDuration="1.725168706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:18:47.72314748 +0000 UTC m=+3365.409225593" watchObservedRunningTime="2026-07-28 15:18:47.725168706 +0000 UTC m=+3365.411246819"
Jul 28 15:18:47.738134 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:47.738091    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" podStartSLOduration=1.738080522 podStartE2EDuration="1.738080522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:18:47.73642856 +0000 UTC m=+3365.422506673" watchObservedRunningTime="2026-07-28 15:18:47.738080522 +0000 UTC m=+3365.424158635"
Jul 28 15:18:52.719644 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:52.719615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:18:58.725203 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:58.725173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:18:58.725710 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:18:58.725350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:19:11.370665 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.370628    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"]
Jul 28 15:19:11.374516 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.374489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:11.376499 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.376474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:19:11.380679 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.380653    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"]
Jul 28 15:19:11.487811 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.487780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s\" (UID: \"2deebcf6-25d1-4189-8982-388fcc2e88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:11.588724 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.588680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s\" (UID: \"2deebcf6-25d1-4189-8982-388fcc2e88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:11.589309 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.589286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s\" (UID: \"2deebcf6-25d1-4189-8982-388fcc2e88da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:11.688439 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.688357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:11.835042 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:11.835016    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"]
Jul 28 15:19:11.838212 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:19:11.838178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deebcf6_25d1_4189_8982_388fcc2e88da.slice/crio-f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b WatchSource:0}: Error finding container f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b: Status 404 returned error can't find the container with id f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b
Jul 28 15:19:12.847957 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:12.847867    2577 generic.go:356] "Generic (PLEG): container finished" podID="2deebcf6-25d1-4189-8982-388fcc2e88da" containerID="acaf8fc5553dd96bfc4f9c383a27a5021e17e4b01407f45dcf8677cb47331240" exitCode=0
Jul 28 15:19:12.848368 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:12.847960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s" event={"ID":"2deebcf6-25d1-4189-8982-388fcc2e88da","Type":"ContainerDied","Data":"acaf8fc5553dd96bfc4f9c383a27a5021e17e4b01407f45dcf8677cb47331240"}
Jul 28 15:19:12.848368 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:12.848019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s" event={"ID":"2deebcf6-25d1-4189-8982-388fcc2e88da","Type":"ContainerStarted","Data":"f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b"}
Jul 28 15:19:14.014696 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.014669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:14.112358 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.112261    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca\") pod \"2deebcf6-25d1-4189-8982-388fcc2e88da\" (UID: \"2deebcf6-25d1-4189-8982-388fcc2e88da\") "
Jul 28 15:19:14.112689 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.112662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2deebcf6-25d1-4189-8982-388fcc2e88da" (UID: "2deebcf6-25d1-4189-8982-388fcc2e88da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:14.213764 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.213729    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2deebcf6-25d1-4189-8982-388fcc2e88da-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:14.860280 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.860248    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"
Jul 28 15:19:14.860470 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.860282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s" event={"ID":"2deebcf6-25d1-4189-8982-388fcc2e88da","Type":"ContainerDied","Data":"f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b"}
Jul 28 15:19:14.860470 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:14.860339    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40781e5db085c3b34a33cb59f335784a5dd95763bc7dee88383263c643e3a4b"
Jul 28 15:19:16.233234 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.233191    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:19:16.233724 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.233443    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" podUID="583769d7-6761-4ca1-9181-1ea37a65ca0a" containerName="maas-api" containerID="cri-o://7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c" gracePeriod=30
Jul 28 15:19:16.285737 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.285698    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:19:16.286014 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.285960    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" podUID="26cc415d-b2df-410e-b82d-f69b90b7bf39" containerName="payload-processing" containerID="cri-o://15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f" gracePeriod=30
Jul 28 15:19:16.295375 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.295346    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:19:16.295645 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.295616    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" podUID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" containerName="payload-pre-processing" containerID="cri-o://ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0" gracePeriod=30
Jul 28 15:19:16.406127 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.406087    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"]
Jul 28 15:19:16.408376 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.408351    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-adb651-vg94s"]
Jul 28 15:19:16.601470 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.601442    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:19:16.603686 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.603665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:19:16.617629 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.617518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:19:16.739201 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbrh4\" (UniqueName: \"kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4\") pod \"26cc415d-b2df-410e-b82d-f69b90b7bf39\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") "
Jul 28 15:19:16.739403 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls\") pod \"583769d7-6761-4ca1-9181-1ea37a65ca0a\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") "
Jul 28 15:19:16.739403 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dm6w\" (UniqueName: \"kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w\") pod \"583769d7-6761-4ca1-9181-1ea37a65ca0a\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") "
Jul 28 15:19:16.739403 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcskg\" (UniqueName: \"kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg\") pod \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") "
Jul 28 15:19:16.739403 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume\") pod \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\" (UID: \"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93\") "
Jul 28 15:19:16.739689 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume\") pod \"26cc415d-b2df-410e-b82d-f69b90b7bf39\" (UID: \"26cc415d-b2df-410e-b82d-f69b90b7bf39\") "
Jul 28 15:19:16.739689 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca\") pod \"583769d7-6761-4ca1-9181-1ea37a65ca0a\" (UID: \"583769d7-6761-4ca1-9181-1ea37a65ca0a\") "
Jul 28 15:19:16.739908 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739876    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" (UID: "dd6af3fc-8a0b-450b-9e3d-c27fe7722d93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:16.739990 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.739909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26cc415d-b2df-410e-b82d-f69b90b7bf39" (UID: "26cc415d-b2df-410e-b82d-f69b90b7bf39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:16.740180 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.740155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "583769d7-6761-4ca1-9181-1ea37a65ca0a" (UID: "583769d7-6761-4ca1-9181-1ea37a65ca0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:16.741552 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.741499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w" (OuterVolumeSpecName: "kube-api-access-7dm6w") pod "583769d7-6761-4ca1-9181-1ea37a65ca0a" (UID: "583769d7-6761-4ca1-9181-1ea37a65ca0a"). InnerVolumeSpecName "kube-api-access-7dm6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:16.741683 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.741575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "583769d7-6761-4ca1-9181-1ea37a65ca0a" (UID: "583769d7-6761-4ca1-9181-1ea37a65ca0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:19:16.741683 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.741614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg" (OuterVolumeSpecName: "kube-api-access-dcskg") pod "dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" (UID: "dd6af3fc-8a0b-450b-9e3d-c27fe7722d93"). InnerVolumeSpecName "kube-api-access-dcskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:16.741683 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.741645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4" (OuterVolumeSpecName: "kube-api-access-zbrh4") pod "26cc415d-b2df-410e-b82d-f69b90b7bf39" (UID: "26cc415d-b2df-410e-b82d-f69b90b7bf39"). InnerVolumeSpecName "kube-api-access-zbrh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:16.786579 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.786482    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deebcf6-25d1-4189-8982-388fcc2e88da" path="/var/lib/kubelet/pods/2deebcf6-25d1-4189-8982-388fcc2e88da/volumes"
Jul 28 15:19:16.840381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840342    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840373    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cc415d-b2df-410e-b82d-f69b90b7bf39-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840383    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583769d7-6761-4ca1-9181-1ea37a65ca0a-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840633 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840392    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrh4\" (UniqueName: \"kubernetes.io/projected/26cc415d-b2df-410e-b82d-f69b90b7bf39-kube-api-access-zbrh4\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840633 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840402    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583769d7-6761-4ca1-9181-1ea37a65ca0a-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840633 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840410    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm6w\" (UniqueName: \"kubernetes.io/projected/583769d7-6761-4ca1-9181-1ea37a65ca0a-kube-api-access-7dm6w\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.840633 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.840418    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcskg\" (UniqueName: \"kubernetes.io/projected/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93-kube-api-access-dcskg\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:16.871944 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.871909    2577 generic.go:356] "Generic (PLEG): container finished" podID="26cc415d-b2df-410e-b82d-f69b90b7bf39" containerID="15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f" exitCode=0
Jul 28 15:19:16.872108 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.871981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"
Jul 28 15:19:16.872108 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.871995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" event={"ID":"26cc415d-b2df-410e-b82d-f69b90b7bf39","Type":"ContainerDied","Data":"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"}
Jul 28 15:19:16.872108 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.872073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb" event={"ID":"26cc415d-b2df-410e-b82d-f69b90b7bf39","Type":"ContainerDied","Data":"ea875772cdaa379df0a021c95f00f4da533c5bd0874f3371238349645f2f1e8d"}
Jul 28 15:19:16.872108 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.872104    2577 scope.go:117] "RemoveContainer" containerID="15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"
Jul 28 15:19:16.873327 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.873304    2577 generic.go:356] "Generic (PLEG): container finished" podID="583769d7-6761-4ca1-9181-1ea37a65ca0a" containerID="7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c" exitCode=0
Jul 28 15:19:16.873422 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.873349    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"
Jul 28 15:19:16.873422 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.873385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" event={"ID":"583769d7-6761-4ca1-9181-1ea37a65ca0a","Type":"ContainerDied","Data":"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"}
Jul 28 15:19:16.873572 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.873464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz" event={"ID":"583769d7-6761-4ca1-9181-1ea37a65ca0a","Type":"ContainerDied","Data":"727eb638f84566c696a09fbbe169b99cc9cc5dc3279c9cbb644bc4778126e50c"}
Jul 28 15:19:16.874927 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.874902    2577 generic.go:356] "Generic (PLEG): container finished" podID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" containerID="ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0" exitCode=0
Jul 28 15:19:16.875081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.874932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" event={"ID":"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93","Type":"ContainerDied","Data":"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"}
Jul 28 15:19:16.875081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.874951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"
Jul 28 15:19:16.875081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.874968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt" event={"ID":"dd6af3fc-8a0b-450b-9e3d-c27fe7722d93","Type":"ContainerDied","Data":"556f912f7d2dfbf4fc7fb4b59e0b81a0ebc7ce785c4567c802af7349c18e1d99"}
Jul 28 15:19:16.882244 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.882227    2577 scope.go:117] "RemoveContainer" containerID="15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"
Jul 28 15:19:16.882586 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:19:16.882559    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f\": container with ID starting with 15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f not found: ID does not exist" containerID="15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"
Jul 28 15:19:16.882676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.882597    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f"} err="failed to get container status \"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f\": rpc error: code = NotFound desc = could not find container \"15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f\": container with ID starting with 15c85e834034dae334aef5b7467868d0cbc864a9ac1a3ad0c6a655051321a50f not found: ID does not exist"
Jul 28 15:19:16.882676 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.882618    2577 scope.go:117] "RemoveContainer" containerID="7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"
Jul 28 15:19:16.891209 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.891192    2577 scope.go:117] "RemoveContainer" containerID="7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"
Jul 28 15:19:16.891469 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:19:16.891448    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c\": container with ID starting with 7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c not found: ID does not exist" containerID="7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"
Jul 28 15:19:16.891569 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.891476    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c"} err="failed to get container status \"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c\": rpc error: code = NotFound desc = could not find container \"7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c\": container with ID starting with 7d30949836d8b92342f91add48187915d53d2ded2d1d754566e834a7786e9f9c not found: ID does not exist"
Jul 28 15:19:16.891569 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.891493    2577 scope.go:117] "RemoveContainer" containerID="ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"
Jul 28 15:19:16.892020 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.892001    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:19:16.897408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.897385    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-adb651-6c96d65c5d-rrjcb"]
Jul 28 15:19:16.899934 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.899913    2577 scope.go:117] "RemoveContainer" containerID="ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"
Jul 28 15:19:16.900208 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:19:16.900190    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0\": container with ID starting with ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0 not found: ID does not exist" containerID="ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"
Jul 28 15:19:16.900276 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.900216    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0"} err="failed to get container status \"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0\": rpc error: code = NotFound desc = could not find container \"ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0\": container with ID starting with ac73985fd662733223575f590dabdb8bc7fcc547929e41e61a669279a56b8cf0 not found: ID does not exist"
Jul 28 15:19:16.906891 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.906865    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:19:16.909465 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.909442    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-adb651-5b4586cbc6-cb5jt"]
Jul 28 15:19:16.918679 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.918651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:19:16.922361 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:16.922337    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-adb651-5966fdf896-sl7qz"]
Jul 28 15:19:18.787249 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:18.787220    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26cc415d-b2df-410e-b82d-f69b90b7bf39" path="/var/lib/kubelet/pods/26cc415d-b2df-410e-b82d-f69b90b7bf39/volumes"
Jul 28 15:19:18.787727 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:18.787612    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583769d7-6761-4ca1-9181-1ea37a65ca0a" path="/var/lib/kubelet/pods/583769d7-6761-4ca1-9181-1ea37a65ca0a/volumes"
Jul 28 15:19:18.787926 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:18.787913    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" path="/var/lib/kubelet/pods/dd6af3fc-8a0b-450b-9e3d-c27fe7722d93/volumes"
Jul 28 15:19:26.911731 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.911679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"]
Jul 28 15:19:26.912227 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912205    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583769d7-6761-4ca1-9181-1ea37a65ca0a" containerName="maas-api"
Jul 28 15:19:26.912227 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912219    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="583769d7-6761-4ca1-9181-1ea37a65ca0a" containerName="maas-api"
Jul 28 15:19:26.912227 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912228    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" containerName="payload-pre-processing"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912237    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" containerName="payload-pre-processing"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912251    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deebcf6-25d1-4189-8982-388fcc2e88da" containerName="revoke-keys"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912260    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deebcf6-25d1-4189-8982-388fcc2e88da" containerName="revoke-keys"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912274    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cc415d-b2df-410e-b82d-f69b90b7bf39" containerName="payload-processing"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912280    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cc415d-b2df-410e-b82d-f69b90b7bf39" containerName="payload-processing"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912380    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6af3fc-8a0b-450b-9e3d-c27fe7722d93" containerName="payload-pre-processing"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912394    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="583769d7-6761-4ca1-9181-1ea37a65ca0a" containerName="maas-api"
Jul 28 15:19:26.912408 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="26cc415d-b2df-410e-b82d-f69b90b7bf39" containerName="payload-processing"
Jul 28 15:19:26.912857 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.912416    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deebcf6-25d1-4189-8982-388fcc2e88da" containerName="revoke-keys"
Jul 28 15:19:26.915771 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.915751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:26.917838 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.917813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:19:26.922439 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:26.922410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"]
Jul 28 15:19:27.042941 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.042905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr\" (UID: \"f3ec93f7-c933-4272-b532-80546e497021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:27.143638 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.143603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr\" (UID: \"f3ec93f7-c933-4272-b532-80546e497021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:27.144475 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.144452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr\" (UID: \"f3ec93f7-c933-4272-b532-80546e497021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:27.230625 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.230593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:27.366117 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.366095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"]
Jul 28 15:19:27.368499 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:19:27.368472    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ec93f7_c933_4272_b532_80546e497021.slice/crio-b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f WatchSource:0}: Error finding container b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f: Status 404 returned error can't find the container with id b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f
Jul 28 15:19:27.940342 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.940240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr" event={"ID":"f3ec93f7-c933-4272-b532-80546e497021","Type":"ContainerStarted","Data":"ea9eadfb3cf992999c1affdc5f234e45cc038091f495b75e1de02b005a1c3867"}
Jul 28 15:19:27.940342 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.940329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr" event={"ID":"f3ec93f7-c933-4272-b532-80546e497021","Type":"ContainerStarted","Data":"b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f"}
Jul 28 15:19:27.955326 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:27.955266    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr" podStartSLOduration=1.955249488 podStartE2EDuration="1.955249488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:19:27.952820553 +0000 UTC m=+3405.638898664" watchObservedRunningTime="2026-07-28 15:19:27.955249488 +0000 UTC m=+3405.641327601"
Jul 28 15:19:28.947589 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:28.947560    2577 generic.go:356] "Generic (PLEG): container finished" podID="f3ec93f7-c933-4272-b532-80546e497021" containerID="ea9eadfb3cf992999c1affdc5f234e45cc038091f495b75e1de02b005a1c3867" exitCode=0
Jul 28 15:19:28.948054 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:28.947705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr" event={"ID":"f3ec93f7-c933-4272-b532-80546e497021","Type":"ContainerDied","Data":"ea9eadfb3cf992999c1affdc5f234e45cc038091f495b75e1de02b005a1c3867"}
Jul 28 15:19:30.095398 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.095372    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:30.173894 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.173854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca\") pod \"f3ec93f7-c933-4272-b532-80546e497021\" (UID: \"f3ec93f7-c933-4272-b532-80546e497021\") "
Jul 28 15:19:30.174293 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.174268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3ec93f7-c933-4272-b532-80546e497021" (UID: "f3ec93f7-c933-4272-b532-80546e497021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:30.274813 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.274778    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ec93f7-c933-4272-b532-80546e497021-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:30.960443 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.960415    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"
Jul 28 15:19:30.960443 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.960426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr" event={"ID":"f3ec93f7-c933-4272-b532-80546e497021","Type":"ContainerDied","Data":"b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f"}
Jul 28 15:19:30.960706 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:30.960475    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60533f109d75e7f1dba9cebb358f9e00eb429a98457d6d66206491dd4de3c6f"
Jul 28 15:19:42.687192 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.687099    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:19:42.687684 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.687389    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" podUID="e339bde1-af9e-4022-9358-618ae503ae63" containerName="maas-api" containerID="cri-o://03c83137519415b0179505fb6f1d7378a148791e0a83892aefcd7358ad4a3ea5" gracePeriod=30
Jul 28 15:19:42.737650 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.737613    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:19:42.737906 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.737853    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" podUID="bc6cecae-6f51-4ab1-840d-df68714dabf8" containerName="payload-processing" containerID="cri-o://5ac1d42751484603a7e5be6593dc3e0faae63e6f5654d47429fd337395b363cc" gracePeriod=30
Jul 28 15:19:42.742152 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.742126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:19:42.742512 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.742352    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerName="payload-pre-processing" containerID="cri-o://acf7823eb38f0e9decbcb3820c807b4b7443c48ad225f414bbfc039a1ffe46df" gracePeriod=30
Jul 28 15:19:42.860200 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.860154    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"]
Jul 28 15:19:42.863294 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:42.863266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-968c04-mzmxr"]
Jul 28 15:19:43.028655 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.028628    2577 generic.go:356] "Generic (PLEG): container finished" podID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerID="acf7823eb38f0e9decbcb3820c807b4b7443c48ad225f414bbfc039a1ffe46df" exitCode=0
Jul 28 15:19:43.028809 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.028701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" event={"ID":"69d438cb-6502-4598-b4d5-b3e2f60b019f","Type":"ContainerDied","Data":"acf7823eb38f0e9decbcb3820c807b4b7443c48ad225f414bbfc039a1ffe46df"}
Jul 28 15:19:43.030202 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.030180    2577 generic.go:356] "Generic (PLEG): container finished" podID="e339bde1-af9e-4022-9358-618ae503ae63" containerID="03c83137519415b0179505fb6f1d7378a148791e0a83892aefcd7358ad4a3ea5" exitCode=0
Jul 28 15:19:43.030320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.030225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" event={"ID":"e339bde1-af9e-4022-9358-618ae503ae63","Type":"ContainerDied","Data":"03c83137519415b0179505fb6f1d7378a148791e0a83892aefcd7358ad4a3ea5"}
Jul 28 15:19:43.030320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.030271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk" event={"ID":"e339bde1-af9e-4022-9358-618ae503ae63","Type":"ContainerDied","Data":"75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75"}
Jul 28 15:19:43.030320 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.030295    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75fae89fb081a1b6d1b6dc99e6c786d1db649b0d3dd735cb84f29e28b18d4c75"
Jul 28 15:19:43.031683 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.031660    2577 generic.go:356] "Generic (PLEG): container finished" podID="bc6cecae-6f51-4ab1-840d-df68714dabf8" containerID="5ac1d42751484603a7e5be6593dc3e0faae63e6f5654d47429fd337395b363cc" exitCode=0
Jul 28 15:19:43.031790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.031727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" event={"ID":"bc6cecae-6f51-4ab1-840d-df68714dabf8","Type":"ContainerDied","Data":"5ac1d42751484603a7e5be6593dc3e0faae63e6f5654d47429fd337395b363cc"}
Jul 28 15:19:43.031790 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.031768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq" event={"ID":"bc6cecae-6f51-4ab1-840d-df68714dabf8","Type":"ContainerDied","Data":"ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9"}
Jul 28 15:19:43.031880 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.031792    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca429c2c38801aebbdc1df60f8e7ff5ca69c258656dc3638945039425e3d9ed9"
Jul 28 15:19:43.033070 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.033051    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:19:43.036481 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.036464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:19:43.057555 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.057499    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:19:43.102305 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102261    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") pod \"e339bde1-af9e-4022-9358-618ae503ae63\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") "
Jul 28 15:19:43.102524 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume\") pod \"69d438cb-6502-4598-b4d5-b3e2f60b019f\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") "
Jul 28 15:19:43.102524 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64v4h\" (UniqueName: \"kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h\") pod \"e339bde1-af9e-4022-9358-618ae503ae63\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") "
Jul 28 15:19:43.102524 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9q4m\" (UniqueName: \"kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m\") pod \"69d438cb-6502-4598-b4d5-b3e2f60b019f\" (UID: \"69d438cb-6502-4598-b4d5-b3e2f60b019f\") "
Jul 28 15:19:43.102524 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd77t\" (UniqueName: \"kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t\") pod \"bc6cecae-6f51-4ab1-840d-df68714dabf8\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") "
Jul 28 15:19:43.102794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume\") pod \"bc6cecae-6f51-4ab1-840d-df68714dabf8\" (UID: \"bc6cecae-6f51-4ab1-840d-df68714dabf8\") "
Jul 28 15:19:43.102794 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.102608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca\") pod \"e339bde1-af9e-4022-9358-618ae503ae63\" (UID: \"e339bde1-af9e-4022-9358-618ae503ae63\") "
Jul 28 15:19:43.103366 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.103337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e339bde1-af9e-4022-9358-618ae503ae63" (UID: "e339bde1-af9e-4022-9358-618ae503ae63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:43.104485 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.104454    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69d438cb-6502-4598-b4d5-b3e2f60b019f" (UID: "69d438cb-6502-4598-b4d5-b3e2f60b019f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:43.104695 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.104575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc6cecae-6f51-4ab1-840d-df68714dabf8" (UID: "bc6cecae-6f51-4ab1-840d-df68714dabf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:43.106192 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.106168    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e339bde1-af9e-4022-9358-618ae503ae63" (UID: "e339bde1-af9e-4022-9358-618ae503ae63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:19:43.106423 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.106396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h" (OuterVolumeSpecName: "kube-api-access-64v4h") pod "e339bde1-af9e-4022-9358-618ae503ae63" (UID: "e339bde1-af9e-4022-9358-618ae503ae63"). InnerVolumeSpecName "kube-api-access-64v4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:43.106813 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.106781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t" (OuterVolumeSpecName: "kube-api-access-sd77t") pod "bc6cecae-6f51-4ab1-840d-df68714dabf8" (UID: "bc6cecae-6f51-4ab1-840d-df68714dabf8"). InnerVolumeSpecName "kube-api-access-sd77t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:43.106923 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.106812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m" (OuterVolumeSpecName: "kube-api-access-f9q4m") pod "69d438cb-6502-4598-b4d5-b3e2f60b019f" (UID: "69d438cb-6502-4598-b4d5-b3e2f60b019f"). InnerVolumeSpecName "kube-api-access-f9q4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:43.203979 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203889    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e339bde1-af9e-4022-9358-618ae503ae63-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.203979 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203930    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d438cb-6502-4598-b4d5-b3e2f60b019f-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.203979 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203946    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64v4h\" (UniqueName: \"kubernetes.io/projected/e339bde1-af9e-4022-9358-618ae503ae63-kube-api-access-64v4h\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.203979 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203961    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9q4m\" (UniqueName: \"kubernetes.io/projected/69d438cb-6502-4598-b4d5-b3e2f60b019f-kube-api-access-f9q4m\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.203979 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203974    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd77t\" (UniqueName: \"kubernetes.io/projected/bc6cecae-6f51-4ab1-840d-df68714dabf8-kube-api-access-sd77t\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.204294 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203986    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6cecae-6f51-4ab1-840d-df68714dabf8-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:43.204294 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:43.203999    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339bde1-af9e-4022-9358-618ae503ae63-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:19:44.045880 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.045849    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"
Jul 28 15:19:44.046380 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.045874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz" event={"ID":"69d438cb-6502-4598-b4d5-b3e2f60b019f","Type":"ContainerDied","Data":"ef0e3074ed11fe2b1f8d89604b8442fa3f97d74ef98934ad464c493502a2d4e2"}
Jul 28 15:19:44.046380 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.045855    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"
Jul 28 15:19:44.046380 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.045978    2577 scope.go:117] "RemoveContainer" containerID="acf7823eb38f0e9decbcb3820c807b4b7443c48ad225f414bbfc039a1ffe46df"
Jul 28 15:19:44.046380 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.046136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"
Jul 28 15:19:44.071232 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.071203    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:19:44.074745 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.074722    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-968c04-698bdfc7b8-49xvk"]
Jul 28 15:19:44.086147 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.086119    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:19:44.092136 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.092112    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-968c04-686cbdc858-884gq"]
Jul 28 15:19:44.100788 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.100765    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:19:44.104193 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.104173    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-968c04-5b49844fb4-88djz"]
Jul 28 15:19:44.785728 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.785691    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" path="/var/lib/kubelet/pods/69d438cb-6502-4598-b4d5-b3e2f60b019f/volumes"
Jul 28 15:19:44.786160 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.786142    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6cecae-6f51-4ab1-840d-df68714dabf8" path="/var/lib/kubelet/pods/bc6cecae-6f51-4ab1-840d-df68714dabf8/volumes"
Jul 28 15:19:44.786581 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.786564    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e339bde1-af9e-4022-9358-618ae503ae63" path="/var/lib/kubelet/pods/e339bde1-af9e-4022-9358-618ae503ae63/volumes"
Jul 28 15:19:44.786916 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:19:44.786900    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ec93f7-c933-4272-b532-80546e497021" path="/var/lib/kubelet/pods/f3ec93f7-c933-4272-b532-80546e497021/volumes"
Jul 28 15:20:32.247934 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.247891    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"]
Jul 28 15:20:32.248489 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248468    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e339bde1-af9e-4022-9358-618ae503ae63" containerName="maas-api"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248493    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e339bde1-af9e-4022-9358-618ae503ae63" containerName="maas-api"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerName="payload-pre-processing"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerName="payload-pre-processing"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248577    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6cecae-6f51-4ab1-840d-df68714dabf8" containerName="payload-processing"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248586    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6cecae-6f51-4ab1-840d-df68714dabf8" containerName="payload-processing"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248596    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ec93f7-c933-4272-b532-80546e497021" containerName="revoke-keys"
Jul 28 15:20:32.248613 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248605    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ec93f7-c933-4272-b532-80546e497021" containerName="revoke-keys"
Jul 28 15:20:32.248965 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248715    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d438cb-6502-4598-b4d5-b3e2f60b019f" containerName="payload-pre-processing"
Jul 28 15:20:32.248965 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248726    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6cecae-6f51-4ab1-840d-df68714dabf8" containerName="payload-processing"
Jul 28 15:20:32.248965 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248737    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e339bde1-af9e-4022-9358-618ae503ae63" containerName="maas-api"
Jul 28 15:20:32.248965 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.248748    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ec93f7-c933-4272-b532-80546e497021" containerName="revoke-keys"
Jul 28 15:20:32.252052 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.252031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:32.254577 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.254529    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:20:32.257689 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.257664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"]
Jul 28 15:20:32.363645 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.363605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl\" (UID: \"b7638726-371e-4d70-8777-2060dbf93685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:32.465053 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.465017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl\" (UID: \"b7638726-371e-4d70-8777-2060dbf93685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:32.465731 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.465703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl\" (UID: \"b7638726-371e-4d70-8777-2060dbf93685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:32.565438 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.565350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:32.911753 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:32.911727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"]
Jul 28 15:20:32.914376 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:20:32.914335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7638726_371e_4d70_8777_2060dbf93685.slice/crio-220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df WatchSource:0}: Error finding container 220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df: Status 404 returned error can't find the container with id 220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df
Jul 28 15:20:33.293842 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:33.293805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl" event={"ID":"b7638726-371e-4d70-8777-2060dbf93685","Type":"ContainerStarted","Data":"f6e69447aa0c29cb8baed0b4cf2ef52f1eb4fe8e4f1e94de41f4fcb65b73d390"}
Jul 28 15:20:33.294257 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:33.293868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl" event={"ID":"b7638726-371e-4d70-8777-2060dbf93685","Type":"ContainerStarted","Data":"220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df"}
Jul 28 15:20:33.308834 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:33.308780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl" podStartSLOduration=1.308763021 podStartE2EDuration="1.308763021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:20:33.308013514 +0000 UTC m=+3470.994091628" watchObservedRunningTime="2026-07-28 15:20:33.308763021 +0000 UTC m=+3470.994841134"
Jul 28 15:20:34.300068 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:34.300032    2577 generic.go:356] "Generic (PLEG): container finished" podID="b7638726-371e-4d70-8777-2060dbf93685" containerID="f6e69447aa0c29cb8baed0b4cf2ef52f1eb4fe8e4f1e94de41f4fcb65b73d390" exitCode=0
Jul 28 15:20:34.300525 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:34.300074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl" event={"ID":"b7638726-371e-4d70-8777-2060dbf93685","Type":"ContainerDied","Data":"f6e69447aa0c29cb8baed0b4cf2ef52f1eb4fe8e4f1e94de41f4fcb65b73d390"}
Jul 28 15:20:35.441341 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:35.441317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:35.492288 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:35.492252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca\") pod \"b7638726-371e-4d70-8777-2060dbf93685\" (UID: \"b7638726-371e-4d70-8777-2060dbf93685\") "
Jul 28 15:20:35.492693 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:35.492669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7638726-371e-4d70-8777-2060dbf93685" (UID: "b7638726-371e-4d70-8777-2060dbf93685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:20:35.593749 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:35.593658    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7638726-371e-4d70-8777-2060dbf93685-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:20:36.311155 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:36.311118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl" event={"ID":"b7638726-371e-4d70-8777-2060dbf93685","Type":"ContainerDied","Data":"220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df"}
Jul 28 15:20:36.311155 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:36.311152    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:36.311422 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:36.311168    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220d110fb18cbe636b938b9359c39019522c0aeed5e3df26e92eeada344261df"
Jul 28 15:20:42.777669 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:42.777603    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"]
Jul 28 15:20:42.784296 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:20:42.784247    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl\" is forbidden: User \"system:node:ip-10-0-142-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-230.ec2.internal' and this object" podUID="b7638726-371e-4d70-8777-2060dbf93685" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"
Jul 28 15:20:42.787719 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:42.787693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-86ef5d-8zzhl"]
Jul 28 15:20:44.785759 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:44.785725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7638726-371e-4d70-8777-2060dbf93685" path="/var/lib/kubelet/pods/b7638726-371e-4d70-8777-2060dbf93685/volumes"
Jul 28 15:20:49.946963 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.946918    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"]
Jul 28 15:20:49.947463 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.947334    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7638726-371e-4d70-8777-2060dbf93685" containerName="revoke-keys"
Jul 28 15:20:49.947463 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.947345    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7638726-371e-4d70-8777-2060dbf93685" containerName="revoke-keys"
Jul 28 15:20:49.947463 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.947418    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7638726-371e-4d70-8777-2060dbf93685" containerName="revoke-keys"
Jul 28 15:20:49.951991 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.951967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:49.954178 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.954155    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:20:49.956117 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:49.956074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"]
Jul 28 15:20:50.025783 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:50.025750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598\" (UID: \"a836543f-9902-4c90-bae2-7ad6de1b0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:50.126437 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:50.126405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598\" (UID: \"a836543f-9902-4c90-bae2-7ad6de1b0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:50.127024 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:50.127006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598\" (UID: \"a836543f-9902-4c90-bae2-7ad6de1b0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:50.265669 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:50.265632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:50.606367 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:50.606287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"]
Jul 28 15:20:50.609354 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:20:50.609323    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda836543f_9902_4c90_bae2_7ad6de1b0bf3.slice/crio-520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b WatchSource:0}: Error finding container 520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b: Status 404 returned error can't find the container with id 520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b
Jul 28 15:20:51.392223 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:51.392175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598" event={"ID":"a836543f-9902-4c90-bae2-7ad6de1b0bf3","Type":"ContainerStarted","Data":"3fb56ed3f4ba31fd403c519e1bef5af60016dc90d2004e619d6c38aa15d26c78"}
Jul 28 15:20:51.392693 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:51.392244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598" event={"ID":"a836543f-9902-4c90-bae2-7ad6de1b0bf3","Type":"ContainerStarted","Data":"520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b"}
Jul 28 15:20:51.406370 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:51.406314    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598" podStartSLOduration=2.406294282 podStartE2EDuration="2.406294282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:20:51.405811098 +0000 UTC m=+3489.091889216" watchObservedRunningTime="2026-07-28 15:20:51.406294282 +0000 UTC m=+3489.092372397"
Jul 28 15:20:52.399078 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:52.399036    2577 generic.go:356] "Generic (PLEG): container finished" podID="a836543f-9902-4c90-bae2-7ad6de1b0bf3" containerID="3fb56ed3f4ba31fd403c519e1bef5af60016dc90d2004e619d6c38aa15d26c78" exitCode=0
Jul 28 15:20:52.399723 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:52.399112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598" event={"ID":"a836543f-9902-4c90-bae2-7ad6de1b0bf3","Type":"ContainerDied","Data":"3fb56ed3f4ba31fd403c519e1bef5af60016dc90d2004e619d6c38aa15d26c78"}
Jul 28 15:20:53.537489 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:53.537464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:53.661185 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:53.661087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca\") pod \"a836543f-9902-4c90-bae2-7ad6de1b0bf3\" (UID: \"a836543f-9902-4c90-bae2-7ad6de1b0bf3\") "
Jul 28 15:20:53.661477 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:53.661455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a836543f-9902-4c90-bae2-7ad6de1b0bf3" (UID: "a836543f-9902-4c90-bae2-7ad6de1b0bf3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:20:53.762792 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:53.762757    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a836543f-9902-4c90-bae2-7ad6de1b0bf3-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:20:54.410025 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:54.409996    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"
Jul 28 15:20:54.410025 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:54.410016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598" event={"ID":"a836543f-9902-4c90-bae2-7ad6de1b0bf3","Type":"ContainerDied","Data":"520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b"}
Jul 28 15:20:54.410295 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:20:54.410063    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="520c5bd315f91c583a7f4ff62cd9e95d737343a229f143dae2b8041725ab8a6b"
Jul 28 15:21:00.317600 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.317561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:21:00.318052 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.317795    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" podUID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" containerName="maas-api" containerID="cri-o://2037f539d5b00afcdd7686d2d3f3a7c3af4bd24964a5663534d925286de777fd" gracePeriod=30
Jul 28 15:21:00.372150 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.372119    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:21:00.372344 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.372320    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" podUID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" containerName="payload-processing" containerID="cri-o://794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6" gracePeriod=30
Jul 28 15:21:00.379075 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.379044    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:21:00.444139 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.444109    2577 generic.go:356] "Generic (PLEG): container finished" podID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" containerID="2037f539d5b00afcdd7686d2d3f3a7c3af4bd24964a5663534d925286de777fd" exitCode=0
Jul 28 15:21:00.444374 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.444185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" event={"ID":"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f","Type":"ContainerDied","Data":"2037f539d5b00afcdd7686d2d3f3a7c3af4bd24964a5663534d925286de777fd"}
Jul 28 15:21:00.444374 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.444326    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" podUID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" containerName="payload-pre-processing" containerID="cri-o://ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046" gracePeriod=30
Jul 28 15:21:00.497755 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.497717    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"]
Jul 28 15:21:00.500549 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.500506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd42ef-rj598"]
Jul 28 15:21:00.653191 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.653162    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:21:00.712550 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.712516    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:21:00.754670 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.754641    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:21:00.785220 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.785179    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a836543f-9902-4c90-bae2-7ad6de1b0bf3" path="/var/lib/kubelet/pods/a836543f-9902-4c90-bae2-7ad6de1b0bf3/volumes"
Jul 28 15:21:00.826528 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.826496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls\") pod \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") "
Jul 28 15:21:00.826726 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.826560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdr9\" (UniqueName: \"kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9\") pod \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") "
Jul 28 15:21:00.826726 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.826675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume\") pod \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") "
Jul 28 15:21:00.826726 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.826701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca\") pod \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\" (UID: \"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f\") "
Jul 28 15:21:00.826726 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.826720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f2rs\" (UniqueName: \"kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs\") pod \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\" (UID: \"b96298cd-c845-41c3-9fce-d1b06e7b56fd\") "
Jul 28 15:21:00.827129 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.827104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b96298cd-c845-41c3-9fce-d1b06e7b56fd" (UID: "b96298cd-c845-41c3-9fce-d1b06e7b56fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:00.827221 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.827101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" (UID: "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:00.828779 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.828755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" (UID: "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:21:00.828864 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.828761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9" (OuterVolumeSpecName: "kube-api-access-nsdr9") pod "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" (UID: "3cb78b86-740c-46d8-89b3-1f0a6eb5f88f"). InnerVolumeSpecName "kube-api-access-nsdr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:00.828864 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.828798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs" (OuterVolumeSpecName: "kube-api-access-7f2rs") pod "b96298cd-c845-41c3-9fce-d1b06e7b56fd" (UID: "b96298cd-c845-41c3-9fce-d1b06e7b56fd"). InnerVolumeSpecName "kube-api-access-7f2rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:00.927894 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.927795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdc6j\" (UniqueName: \"kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j\") pod \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") "
Jul 28 15:21:00.927894 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.927858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume\") pod \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\" (UID: \"33948fa9-dd3e-4495-b5c4-d918705fe3ce\") "
Jul 28 15:21:00.928149 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928130    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96298cd-c845-41c3-9fce-d1b06e7b56fd-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:00.928214 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928156    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-openshift-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:00.928214 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928171    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2rs\" (UniqueName: \"kubernetes.io/projected/b96298cd-c845-41c3-9fce-d1b06e7b56fd-kube-api-access-7f2rs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:00.928214 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928187    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-maas-api-tls\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:00.928214 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928200    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdr9\" (UniqueName: \"kubernetes.io/projected/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f-kube-api-access-nsdr9\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:00.928354 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.928208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33948fa9-dd3e-4495-b5c4-d918705fe3ce" (UID: "33948fa9-dd3e-4495-b5c4-d918705fe3ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:00.929889 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:00.929869    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j" (OuterVolumeSpecName: "kube-api-access-qdc6j") pod "33948fa9-dd3e-4495-b5c4-d918705fe3ce" (UID: "33948fa9-dd3e-4495-b5c4-d918705fe3ce"). InnerVolumeSpecName "kube-api-access-qdc6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:01.029473 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.029434    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdc6j\" (UniqueName: \"kubernetes.io/projected/33948fa9-dd3e-4495-b5c4-d918705fe3ce-kube-api-access-qdc6j\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:01.029473 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.029461    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33948fa9-dd3e-4495-b5c4-d918705fe3ce-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:01.450222 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.450191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"
Jul 28 15:21:01.450222 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.450203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf" event={"ID":"3cb78b86-740c-46d8-89b3-1f0a6eb5f88f","Type":"ContainerDied","Data":"64047eca30fe91c926c765737f322f59242a5540fb253e6e71b45227771a27f3"}
Jul 28 15:21:01.450821 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.450266    2577 scope.go:117] "RemoveContainer" containerID="2037f539d5b00afcdd7686d2d3f3a7c3af4bd24964a5663534d925286de777fd"
Jul 28 15:21:01.451694 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.451667    2577 generic.go:356] "Generic (PLEG): container finished" podID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" containerID="794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6" exitCode=0
Jul 28 15:21:01.451931 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.451706    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"
Jul 28 15:21:01.451931 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.451740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" event={"ID":"33948fa9-dd3e-4495-b5c4-d918705fe3ce","Type":"ContainerDied","Data":"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"}
Jul 28 15:21:01.451931 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.451793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx" event={"ID":"33948fa9-dd3e-4495-b5c4-d918705fe3ce","Type":"ContainerDied","Data":"4e766ecae21b49ba24fbc12c800874203c091a036326528b97e38bbcc32f8667"}
Jul 28 15:21:01.453156 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.453131    2577 generic.go:356] "Generic (PLEG): container finished" podID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" containerID="ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046" exitCode=0
Jul 28 15:21:01.453285 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.453201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" event={"ID":"b96298cd-c845-41c3-9fce-d1b06e7b56fd","Type":"ContainerDied","Data":"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"}
Jul 28 15:21:01.453285 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.453237    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"
Jul 28 15:21:01.453285 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.453260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx" event={"ID":"b96298cd-c845-41c3-9fce-d1b06e7b56fd","Type":"ContainerDied","Data":"d9ab65c19a379730ada6fc38e59ff0799218faddf02c5fb207d5cc1d2e93976e"}
Jul 28 15:21:01.460081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.460059    2577 scope.go:117] "RemoveContainer" containerID="794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"
Jul 28 15:21:01.469683 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.469662    2577 scope.go:117] "RemoveContainer" containerID="794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"
Jul 28 15:21:01.469989 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:21:01.469969    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6\": container with ID starting with 794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6 not found: ID does not exist" containerID="794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"
Jul 28 15:21:01.470067 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.469999    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6"} err="failed to get container status \"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6\": rpc error: code = NotFound desc = could not find container \"794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6\": container with ID starting with 794e72c7a5b3ca7792e60d66d3059ec77b2bdb2e80571e348544305888a72ea6 not found: ID does not exist"
Jul 28 15:21:01.470067 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.470017    2577 scope.go:117] "RemoveContainer" containerID="ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"
Jul 28 15:21:01.478076 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.478051    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:21:01.478288 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.478271    2577 scope.go:117] "RemoveContainer" containerID="ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"
Jul 28 15:21:01.478570 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:21:01.478551    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046\": container with ID starting with ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046 not found: ID does not exist" containerID="ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"
Jul 28 15:21:01.478621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.478577    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046"} err="failed to get container status \"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046\": rpc error: code = NotFound desc = could not find container \"ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046\": container with ID starting with ea0292b5fda9be3d3bfb29f054084b83053eb101dc72f281db9d1410de1e9046 not found: ID does not exist"
Jul 28 15:21:01.483674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.483651    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-fd42ef-fd5cc8b8d-j2dnx"]
Jul 28 15:21:01.492821 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.492797    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:21:01.496208 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.496178    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd42ef-6ddc5986c-8t8nf"]
Jul 28 15:21:01.505421 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.505398    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:21:01.507083 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:01.507057    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-fd42ef-6bdd5b485d-m6rzx"]
Jul 28 15:21:02.786053 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:02.786015    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" path="/var/lib/kubelet/pods/33948fa9-dd3e-4495-b5c4-d918705fe3ce/volumes"
Jul 28 15:21:02.786444 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:02.786409    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" path="/var/lib/kubelet/pods/3cb78b86-740c-46d8-89b3-1f0a6eb5f88f/volumes"
Jul 28 15:21:02.786756 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:02.786742    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" path="/var/lib/kubelet/pods/b96298cd-c845-41c3-9fce-d1b06e7b56fd/volumes"
Jul 28 15:21:07.367042 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.366964    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:21:07.367462 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.367234    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy" containerID="cri-o://5c4684609cf21fc12c167fb14c76be9ec02324c0fbffc2bb226ab7441c7593b4" gracePeriod=30
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.724486    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"]
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725311    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" containerName="payload-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725330    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" containerName="payload-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" containerName="maas-api"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725362    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" containerName="maas-api"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725408    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a836543f-9902-4c90-bae2-7ad6de1b0bf3" containerName="revoke-keys"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725418    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a836543f-9902-4c90-bae2-7ad6de1b0bf3" containerName="revoke-keys"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" containerName="payload-pre-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725466    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" containerName="payload-pre-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725629    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb78b86-740c-46d8-89b3-1f0a6eb5f88f" containerName="maas-api"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725647    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33948fa9-dd3e-4495-b5c4-d918705fe3ce" containerName="payload-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725666    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96298cd-c845-41c3-9fce-d1b06e7b56fd" containerName="payload-pre-processing"
Jul 28 15:21:07.728381 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.725678    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a836543f-9902-4c90-bae2-7ad6de1b0bf3" containerName="revoke-keys"
Jul 28 15:21:07.733228 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.733201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:07.733771 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.733657    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"]
Jul 28 15:21:07.736041 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.736018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:21:07.785496 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.785459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7\" (UID: \"f85c2d48-e188-4036-a700-c880b20b5081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:07.886744 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.886708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7\" (UID: \"f85c2d48-e188-4036-a700-c880b20b5081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:07.887357 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:07.887335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7\" (UID: \"f85c2d48-e188-4036-a700-c880b20b5081\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:08.046352 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:08.046263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:08.188030 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:08.187996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"]
Jul 28 15:21:08.189372 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:21:08.189342    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85c2d48_e188_4036_a700_c880b20b5081.slice/crio-813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33 WatchSource:0}: Error finding container 813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33: Status 404 returned error can't find the container with id 813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33
Jul 28 15:21:08.492448 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:08.492406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7" event={"ID":"f85c2d48-e188-4036-a700-c880b20b5081","Type":"ContainerStarted","Data":"813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33"}
Jul 28 15:21:09.499179 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:09.499146    2577 generic.go:356] "Generic (PLEG): container finished" podID="f85c2d48-e188-4036-a700-c880b20b5081" containerID="a54d7c478d4fc4c17db842aa1137504ff0c0cb130a50dca931341464c0a3d6f3" exitCode=0
Jul 28 15:21:09.499583 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:09.499232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7" event={"ID":"f85c2d48-e188-4036-a700-c880b20b5081","Type":"ContainerDied","Data":"a54d7c478d4fc4c17db842aa1137504ff0c0cb130a50dca931341464c0a3d6f3"}
Jul 28 15:21:10.639081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:10.639054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:10.709728 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:10.709699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca\") pod \"f85c2d48-e188-4036-a700-c880b20b5081\" (UID: \"f85c2d48-e188-4036-a700-c880b20b5081\") "
Jul 28 15:21:10.710081 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:10.710057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f85c2d48-e188-4036-a700-c880b20b5081" (UID: "f85c2d48-e188-4036-a700-c880b20b5081"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:10.810594 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:10.810564    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85c2d48-e188-4036-a700-c880b20b5081-maas-api-service-ca\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:11.511230 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:11.511188    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"
Jul 28 15:21:11.511464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:11.511193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7" event={"ID":"f85c2d48-e188-4036-a700-c880b20b5081","Type":"ContainerDied","Data":"813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33"}
Jul 28 15:21:11.511464 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:11.511307    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="813ca9e1c738a2019d7261c2fe83536c466d0ad328457eac39401b7ef57d1e33"
Jul 28 15:21:12.519401 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.519367    2577 generic.go:356] "Generic (PLEG): container finished" podID="03cdc7b5-db51-49e2-8907-487d448723dd" containerID="5c4684609cf21fc12c167fb14c76be9ec02324c0fbffc2bb226ab7441c7593b4" exitCode=0
Jul 28 15:21:12.519830 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.519437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerDied","Data":"5c4684609cf21fc12c167fb14c76be9ec02324c0fbffc2bb226ab7441c7593b4"}
Jul 28 15:21:12.519830 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.519498    2577 scope.go:117] "RemoveContainer" containerID="1529ddb503368923c926fed760a55139b0514b1e61d51d7870f2ebe0253147ae"
Jul 28 15:21:12.633358 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.633291    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:21:12.727702 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727702 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727728    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727818    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nlj5\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.727971 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.727952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert\") pod \"03cdc7b5-db51-49e2-8907-487d448723dd\" (UID: \"03cdc7b5-db51-49e2-8907-487d448723dd\") "
Jul 28 15:21:12.728326 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728105    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:12.728326 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:12.728326 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728314    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.728492 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:12.728492 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:12.728649 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.728627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:12.730117 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.730090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:12.730230 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.730185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:21:12.730451 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.730426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5" (OuterVolumeSpecName: "kube-api-access-4nlj5") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "kube-api-access-4nlj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:12.730552 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.730488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03cdc7b5-db51-49e2-8907-487d448723dd" (UID: "03cdc7b5-db51-49e2-8907-487d448723dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:12.829221 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829188    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03cdc7b5-db51-49e2-8907-487d448723dd-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829221 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829222    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03cdc7b5-db51-49e2-8907-487d448723dd-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829234    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829247    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829258    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829270    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nlj5\" (UniqueName: \"kubernetes.io/projected/03cdc7b5-db51-49e2-8907-487d448723dd-kube-api-access-4nlj5\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829284    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:12.829426 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:12.829295    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03cdc7b5-db51-49e2-8907-487d448723dd-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:13.526179 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:13.526145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w" event={"ID":"03cdc7b5-db51-49e2-8907-487d448723dd","Type":"ContainerDied","Data":"5598eba56b696d7e677884e3065c620c0d07d27be304167cc3663a33e86479fc"}
Jul 28 15:21:13.526653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:13.526234    2577 scope.go:117] "RemoveContainer" containerID="5c4684609cf21fc12c167fb14c76be9ec02324c0fbffc2bb226ab7441c7593b4"
Jul 28 15:21:13.526653 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:13.526160    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"
Jul 28 15:21:13.542458 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:13.542427    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:21:13.546301 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:13.546275    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fd42ef-openshift-default-76549c86bd-b7g7w"]
Jul 28 15:21:14.785742 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:14.785706    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" path="/var/lib/kubelet/pods/03cdc7b5-db51-49e2-8907-487d448723dd/volumes"
Jul 28 15:21:18.144198 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.144162    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:21:18.144813 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.144781    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" podUID="eb3517b2-6d18-4a50-a1aa-dab3534da925" containerName="payload-pre-processing" containerID="cri-o://d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff" gracePeriod=30
Jul 28 15:21:18.277448 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.277410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"]
Jul 28 15:21:18.280619 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.280588    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc713f-v9lj7"]
Jul 28 15:21:18.399991 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.399926    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:21:18.481878 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.481847    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbr5b\" (UniqueName: \"kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b\") pod \"eb3517b2-6d18-4a50-a1aa-dab3534da925\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") "
Jul 28 15:21:18.482054 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.481959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume\") pod \"eb3517b2-6d18-4a50-a1aa-dab3534da925\" (UID: \"eb3517b2-6d18-4a50-a1aa-dab3534da925\") "
Jul 28 15:21:18.482334 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.482313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb3517b2-6d18-4a50-a1aa-dab3534da925" (UID: "eb3517b2-6d18-4a50-a1aa-dab3534da925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:18.484010 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.483978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b" (OuterVolumeSpecName: "kube-api-access-hbr5b") pod "eb3517b2-6d18-4a50-a1aa-dab3534da925" (UID: "eb3517b2-6d18-4a50-a1aa-dab3534da925"). InnerVolumeSpecName "kube-api-access-hbr5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:18.553721 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.553685    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb3517b2-6d18-4a50-a1aa-dab3534da925" containerID="d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff" exitCode=0
Jul 28 15:21:18.553904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.553732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"
Jul 28 15:21:18.553904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.553743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" event={"ID":"eb3517b2-6d18-4a50-a1aa-dab3534da925","Type":"ContainerDied","Data":"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"}
Jul 28 15:21:18.553904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.553796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th" event={"ID":"eb3517b2-6d18-4a50-a1aa-dab3534da925","Type":"ContainerDied","Data":"d42b1e7119133bbec589c2944c85445b1b3120c5ec7cb0557d9da03dad9ec79b"}
Jul 28 15:21:18.553904 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.553830    2577 scope.go:117] "RemoveContainer" containerID="d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"
Jul 28 15:21:18.562427 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.562411    2577 scope.go:117] "RemoveContainer" containerID="d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"
Jul 28 15:21:18.562710 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:21:18.562691    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff\": container with ID starting with d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff not found: ID does not exist" containerID="d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"
Jul 28 15:21:18.562793 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.562718    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff"} err="failed to get container status \"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff\": rpc error: code = NotFound desc = could not find container \"d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff\": container with ID starting with d6732571f0d609869cbf921022fa149d0286b42317a943703ecb35411fcf88ff not found: ID does not exist"
Jul 28 15:21:18.573529 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.573504    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:21:18.576168 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.576139    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-bc713f-5b8b868b7b-847th"]
Jul 28 15:21:18.583090 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.583073    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3517b2-6d18-4a50-a1aa-dab3534da925-plugins-config-volume\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:18.583150 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.583093    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbr5b\" (UniqueName: \"kubernetes.io/projected/eb3517b2-6d18-4a50-a1aa-dab3534da925-kube-api-access-hbr5b\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:18.785584 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.785552    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3517b2-6d18-4a50-a1aa-dab3534da925" path="/var/lib/kubelet/pods/eb3517b2-6d18-4a50-a1aa-dab3534da925/volumes"
Jul 28 15:21:18.785868 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:18.785856    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f85c2d48-e188-4036-a700-c880b20b5081" path="/var/lib/kubelet/pods/f85c2d48-e188-4036-a700-c880b20b5081/volumes"
Jul 28 15:21:25.278203 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:25.278164    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:21:25.278719 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:25.278422    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy" containerID="cri-o://2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430" gracePeriod=30
Jul 28 15:21:30.532252 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.532223    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:21:30.595870 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.595870 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.595870 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf5tq\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.595979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:30.596159 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket\") pod \"d7acabd4-df78-4a83-8d5a-1d379abd5205\" (UID: \"d7acabd4-df78-4a83-8d5a-1d379abd5205\") "
Jul 28 15:21:30.596509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:30.596509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596478    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-certs\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.596509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596496    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-credential-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.596509 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:30.597020 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.596991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:21:30.597102 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.597078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:30.598246 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.598216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:21:30.598346 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.598295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:21:30.598346 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.598325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq" (OuterVolumeSpecName: "kube-api-access-gf5tq") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "kube-api-access-gf5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:30.598431 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.598327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7acabd4-df78-4a83-8d5a-1d379abd5205" (UID: "d7acabd4-df78-4a83-8d5a-1d379abd5205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:21:30.613557 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.613516    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerID="2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430" exitCode=0
Jul 28 15:21:30.613673 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.613565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerDied","Data":"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"}
Jul 28 15:21:30.613673 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.613587    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"
Jul 28 15:21:30.613673 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.613620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26" event={"ID":"d7acabd4-df78-4a83-8d5a-1d379abd5205","Type":"ContainerDied","Data":"e1bd9a2b4a1402bdee67451ac7312504a0a2cc13c55fcb689d1f7ccf60ac819c"}
Jul 28 15:21:30.613673 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.613642    2577 scope.go:117] "RemoveContainer" containerID="2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"
Jul 28 15:21:30.623574 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.623547    2577 scope.go:117] "RemoveContainer" containerID="8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"
Jul 28 15:21:30.632031 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.632007    2577 scope.go:117] "RemoveContainer" containerID="2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"
Jul 28 15:21:30.632340 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:21:30.632321    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430\": container with ID starting with 2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430 not found: ID does not exist" containerID="2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"
Jul 28 15:21:30.632390 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.632352    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430"} err="failed to get container status \"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430\": rpc error: code = NotFound desc = could not find container \"2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430\": container with ID starting with 2a5509fa862396d8799db1277363fb051a15bb1e84fd2486c813be3d00f33430 not found: ID does not exist"
Jul 28 15:21:30.632390 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.632372    2577 scope.go:117] "RemoveContainer" containerID="8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"
Jul 28 15:21:30.632676 ip-10-0-142-230 kubenswrapper[2577]: E0728 15:21:30.632654    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4\": container with ID starting with 8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4 not found: ID does not exist" containerID="8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"
Jul 28 15:21:30.632732 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.632683    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4"} err="failed to get container status \"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4\": rpc error: code = NotFound desc = could not find container \"8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4\": container with ID starting with 8b9592d8afb3d4e303a8249d9e2b01c164810ab4700f39922ec844d03c564dc4 not found: ID does not exist"
Jul 28 15:21:30.644235 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.644208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:21:30.654293 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.654262    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bc713f-openshift-default-77b59c65d4-4zv26"]
Jul 28 15:21:30.697126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697090    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-token\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697119    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-podinfo\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697126 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697128    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5tq\" (UniqueName: \"kubernetes.io/projected/d7acabd4-df78-4a83-8d5a-1d379abd5205-kube-api-access-gf5tq\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697330 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697139    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7acabd4-df78-4a83-8d5a-1d379abd5205-istiod-ca-cert\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697330 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697147    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-envoy\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697330 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697154    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-istio-data\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.697330 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.697161    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7acabd4-df78-4a83-8d5a-1d379abd5205-workload-socket\") on node \"ip-10-0-142-230.ec2.internal\" DevicePath \"\""
Jul 28 15:21:30.786185 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:30.786152    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" path="/var/lib/kubelet/pods/d7acabd4-df78-4a83-8d5a-1d379abd5205/volumes"
Jul 28 15:21:31.650657 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:31.650624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77bfd6665c-f59p8_c149fbb1-76ac-45c5-8e69-5f7ac4e942d3/manager/0.log"
Jul 28 15:21:32.286251 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:32.286219    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-ttblg_4ea6be87-b5f4-491b-bb79-1322bbed538b/copy-manifests/0.log"
Jul 28 15:21:32.292963 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:32.292927    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-ttblg_4ea6be87-b5f4-491b-bb79-1322bbed538b/manager/0.log"
Jul 28 15:21:32.885274 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:32.885240    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754180-nqz7s_e14313d4-8aac-4fc2-b16d-e2b7c30cf6cc/cleanup/0.log"
Jul 28 15:21:32.997235 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:32.997200    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754195-vwfp7_b3f3e510-e0f4-457c-8445-5868db7f7b42/cleanup/0.log"
Jul 28 15:21:33.106767 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:33.106732    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-cj5nj_b37987e4-b840-40b6-b8e7-64e929b7a173/postgres/0.log"
Jul 28 15:21:34.766235 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:34.766207    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cfkc9_577da352-7224-47f7-855d-dbcece2dc0f3/kuadrant-console-plugin/0.log"
Jul 28 15:21:35.951439 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:35.951408    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-k8m7s_d3f0aed6-a460-4914-a4f0-f87dbf300ba1/kube-auth-proxy/0.log"
Jul 28 15:21:36.306980 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:36.306946    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:21:36.420557 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:36.420506    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757957ffc-frc2z_3300c232-216d-424b-9ce2-7220f6dbf8a3/router/0.log"
Jul 28 15:21:36.763276 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:36.763247    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jthfl_bfbf0720-8d09-4fdb-95e9-f9f60c71750d/main/0.log"
Jul 28 15:21:37.213239 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:37.213215    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zv5z7_fecd59e1-4976-40ba-8231-0142b390b5f9/main/0.log"
Jul 28 15:21:43.201119 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9w7l5/must-gather-65l6h"]
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201464    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201476    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201500    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201506    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201520    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f85c2d48-e188-4036-a700-c880b20b5081" containerName="revoke-keys"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201526    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85c2d48-e188-4036-a700-c880b20b5081" containerName="revoke-keys"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201553    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3517b2-6d18-4a50-a1aa-dab3534da925" containerName="payload-pre-processing"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201558    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3517b2-6d18-4a50-a1aa-dab3534da925" containerName="payload-pre-processing"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201614    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f85c2d48-e188-4036-a700-c880b20b5081" containerName="revoke-keys"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201621    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.201621 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201628    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3517b2-6d18-4a50-a1aa-dab3534da925" containerName="payload-pre-processing"
Jul 28 15:21:43.202079 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201635    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.202079 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201641    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.202079 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201646    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7acabd4-df78-4a83-8d5a-1d379abd5205" containerName="istio-proxy"
Jul 28 15:21:43.202079 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201719    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.202079 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.201725    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="03cdc7b5-db51-49e2-8907-487d448723dd" containerName="istio-proxy"
Jul 28 15:21:43.206460 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.206432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.209089 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.209054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9w7l5\"/\"kube-root-ca.crt\""
Jul 28 15:21:43.209248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.209103    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9w7l5\"/\"openshift-service-ca.crt\""
Jul 28 15:21:43.209248 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.209131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9w7l5\"/\"default-dockercfg-krzlf\""
Jul 28 15:21:43.210972 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.210947    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/must-gather-65l6h"]
Jul 28 15:21:43.317836 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.317800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcd8\" (UniqueName: \"kubernetes.io/projected/e1cb90dd-63cf-4133-bdc3-849824ad43cb-kube-api-access-xjcd8\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.318029 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.317847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1cb90dd-63cf-4133-bdc3-849824ad43cb-must-gather-output\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.418786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.418750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcd8\" (UniqueName: \"kubernetes.io/projected/e1cb90dd-63cf-4133-bdc3-849824ad43cb-kube-api-access-xjcd8\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.418978 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.418799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1cb90dd-63cf-4133-bdc3-849824ad43cb-must-gather-output\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.419123 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.419105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e1cb90dd-63cf-4133-bdc3-849824ad43cb-must-gather-output\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.428845 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.428817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcd8\" (UniqueName: \"kubernetes.io/projected/e1cb90dd-63cf-4133-bdc3-849824ad43cb-kube-api-access-xjcd8\") pod \"must-gather-65l6h\" (UID: \"e1cb90dd-63cf-4133-bdc3-849824ad43cb\") " pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.517314 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.517280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/must-gather-65l6h"
Jul 28 15:21:43.651816 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.651744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/must-gather-65l6h"]
Jul 28 15:21:43.654052 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:21:43.654024    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1cb90dd_63cf_4133_bdc3_849824ad43cb.slice/crio-43f8ec01fc54f1a31d9ae2077faf5eba77073db9c9d82dadf67e21118e7bd099 WatchSource:0}: Error finding container 43f8ec01fc54f1a31d9ae2077faf5eba77073db9c9d82dadf67e21118e7bd099: Status 404 returned error can't find the container with id 43f8ec01fc54f1a31d9ae2077faf5eba77073db9c9d82dadf67e21118e7bd099
Jul 28 15:21:43.680411 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:43.680378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/must-gather-65l6h" event={"ID":"e1cb90dd-63cf-4133-bdc3-849824ad43cb","Type":"ContainerStarted","Data":"43f8ec01fc54f1a31d9ae2077faf5eba77073db9c9d82dadf67e21118e7bd099"}
Jul 28 15:21:45.695145 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:45.695097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/must-gather-65l6h" event={"ID":"e1cb90dd-63cf-4133-bdc3-849824ad43cb","Type":"ContainerStarted","Data":"86e5cb02112931f981d272c43cbfe5642fc177a088f013ae06bb4bd75c8796df"}
Jul 28 15:21:45.695694 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:45.695180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/must-gather-65l6h" event={"ID":"e1cb90dd-63cf-4133-bdc3-849824ad43cb","Type":"ContainerStarted","Data":"3abdac949c63481aa0b430ce9758bac0b6591fffefbbca14fb4d26c0c55565e8"}
Jul 28 15:21:45.710938 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:45.710888    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9w7l5/must-gather-65l6h" podStartSLOduration=1.4413395119999999 podStartE2EDuration="2.710871018s" totalImagesPullingTime="1.269531506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:21:45.709902952 +0000 UTC m=+3543.395981065" watchObservedRunningTime="2026-07-28 15:21:45.710871018 +0000 UTC m=+3543.396949129"
Jul 28 15:21:46.470145 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:46.470111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4fzdm_3fe758b3-c1a0-4e29-a50f-ce4b9c06dd8b/global-pull-secret-syncer/0.log"
Jul 28 15:21:46.660887 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:46.660857    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgnhf_c0cdcf6e-c202-43bb-8653-31f0c22b0e5a/konnectivity-agent/0.log"
Jul 28 15:21:46.728942 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:46.728864    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-230.ec2.internal_206bac433f2fce4d8e4d8f2249cfb20b/haproxy/0.log"
Jul 28 15:21:51.292778 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:51.292745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cfkc9_577da352-7224-47f7-855d-dbcece2dc0f3/kuadrant-console-plugin/0.log"
Jul 28 15:21:53.401047 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.400440    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jnzrh_1af3ef47-00b0-4639-9d94-f99cf0334097/kube-state-metrics/0.log"
Jul 28 15:21:53.427692 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.427624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jnzrh_1af3ef47-00b0-4639-9d94-f99cf0334097/kube-rbac-proxy-main/0.log"
Jul 28 15:21:53.447876 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.447844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jnzrh_1af3ef47-00b0-4639-9d94-f99cf0334097/kube-rbac-proxy-self/0.log"
Jul 28 15:21:53.689062 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.689025    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5vxj_3eee1f1f-681b-41d3-8d3f-0b9990289f97/node-exporter/0.log"
Jul 28 15:21:53.712891 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.712862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5vxj_3eee1f1f-681b-41d3-8d3f-0b9990289f97/kube-rbac-proxy/0.log"
Jul 28 15:21:53.741982 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.741953    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r5vxj_3eee1f1f-681b-41d3-8d3f-0b9990289f97/init-textfile/0.log"
Jul 28 15:21:53.885958 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.885897    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/prometheus/0.log"
Jul 28 15:21:53.910576 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.910520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/config-reloader/0.log"
Jul 28 15:21:53.931484 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.931379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/thanos-sidecar/0.log"
Jul 28 15:21:53.954557 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.954484    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/kube-rbac-proxy-web/0.log"
Jul 28 15:21:53.983179 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:53.983102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/kube-rbac-proxy/0.log"
Jul 28 15:21:54.007747 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:54.007703    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/kube-rbac-proxy-thanos/0.log"
Jul 28 15:21:54.038162 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:54.038093    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8500d7c-2bce-4cf9-958b-8d98959f0d9d/init-config-reloader/0.log"
Jul 28 15:21:54.071767 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:54.071728    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-88s9g_6bb46449-8486-46b3-9ab2-5c1b3f52ad38/prometheus-operator/0.log"
Jul 28 15:21:54.103894 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:54.103855    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-88s9g_6bb46449-8486-46b3-9ab2-5c1b3f52ad38/kube-rbac-proxy/0.log"
Jul 28 15:21:55.856069 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:55.856037    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-46766_626a50b6-d25d-45ee-925a-ac17fa5e8990/networking-console-plugin/0.log"
Jul 28 15:21:56.457396 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:56.457365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/2.log"
Jul 28 15:21:56.463649 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:56.463617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2mw5m_ef95864b-dd12-48f1-b4f7-69eb0e2d0f10/console-operator/3.log"
Jul 28 15:21:57.755390 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.755344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"]
Jul 28 15:21:57.763351 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.763317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.768194 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.768157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"]
Jul 28 15:21:57.882560 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.882061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-lib-modules\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.882560 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.882147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-podres\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.882560 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.882218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmpv\" (UniqueName: \"kubernetes.io/projected/4b41ff86-32f4-4c9c-a51d-f49e8586961e-kube-api-access-vhmpv\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.882560 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.882257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-proc\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.882560 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.882300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-sys\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983634 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-podres\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmpv\" (UniqueName: \"kubernetes.io/projected/4b41ff86-32f4-4c9c-a51d-f49e8586961e-kube-api-access-vhmpv\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-proc\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-sys\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-lib-modules\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-podres\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.983835 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-proc\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.984131 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-sys\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.984131 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.983955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b41ff86-32f4-4c9c-a51d-f49e8586961e-lib-modules\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:57.992786 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:57.992727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmpv\" (UniqueName: \"kubernetes.io/projected/4b41ff86-32f4-4c9c-a51d-f49e8586961e-kube-api-access-vhmpv\") pod \"perf-node-gather-daemonset-56kcz\" (UID: \"4b41ff86-32f4-4c9c-a51d-f49e8586961e\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:58.100369 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.100105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:58.275289 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.275246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"]
Jul 28 15:21:58.281282 ip-10-0-142-230 kubenswrapper[2577]: W0728 15:21:58.281238    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b41ff86_32f4_4c9c_a51d_f49e8586961e.slice/crio-9fb53fd392b309427363512683799fed60d16f8f8dd59468943d0740c1573155 WatchSource:0}: Error finding container 9fb53fd392b309427363512683799fed60d16f8f8dd59468943d0740c1573155: Status 404 returned error can't find the container with id 9fb53fd392b309427363512683799fed60d16f8f8dd59468943d0740c1573155
Jul 28 15:21:58.799162 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.799129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz" event={"ID":"4b41ff86-32f4-4c9c-a51d-f49e8586961e","Type":"ContainerStarted","Data":"3a716a76f2a3863344c02f1103d58ea5a621690b94243fe0966c3a67df0c74a3"}
Jul 28 15:21:58.799604 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.799192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz" event={"ID":"4b41ff86-32f4-4c9c-a51d-f49e8586961e","Type":"ContainerStarted","Data":"9fb53fd392b309427363512683799fed60d16f8f8dd59468943d0740c1573155"}
Jul 28 15:21:58.799604 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.799221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:21:58.815742 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.815680    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz" podStartSLOduration=1.815657946 podStartE2EDuration="1.815657946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:21:58.813468234 +0000 UTC m=+3556.499546349" watchObservedRunningTime="2026-07-28 15:21:58.815657946 +0000 UTC m=+3556.501736061"
Jul 28 15:21:58.857419 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.857384    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9chw8_3529d4aa-3f68-4339-824d-dbfc4872c9ce/dns/0.log"
Jul 28 15:21:58.875364 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:58.875333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9chw8_3529d4aa-3f68-4339-824d-dbfc4872c9ce/kube-rbac-proxy/0.log"
Jul 28 15:21:59.007650 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:59.007618    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jntvr_d0c388cc-d3a8-4cee-aea7-8eadddae330e/dns-node-resolver/0.log"
Jul 28 15:21:59.580644 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:21:59.580616    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ckjp9_a9641f09-cad8-4bde-9109-1a7da203da71/node-ca/0.log"
Jul 28 15:22:00.750044 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:00.750015    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-k8m7s_d3f0aed6-a460-4914-a4f0-f87dbf300ba1/kube-auth-proxy/0.log"
Jul 28 15:22:00.861762 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:00.861736    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:22:00.882421 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:00.882392    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757957ffc-frc2z_3300c232-216d-424b-9ce2-7220f6dbf8a3/router/0.log"
Jul 28 15:22:01.730050 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:01.730020    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m7z6c_958f61c6-5bc6-42e7-ac00-c6e86ec3d35c/serve-healthcheck-canary/0.log"
Jul 28 15:22:02.185110 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:02.185021    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-skv6z_ddbd6182-8e34-4f55-9518-56d442d0c058/insights-operator/0.log"
Jul 28 15:22:02.186071 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:02.186049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-skv6z_ddbd6182-8e34-4f55-9518-56d442d0c058/insights-operator/1.log"
Jul 28 15:22:02.333503 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:02.333460    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zsnpr_103f1358-62b6-466c-a0a5-61007d6c4506/kube-rbac-proxy/0.log"
Jul 28 15:22:02.353647 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:02.353614    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zsnpr_103f1358-62b6-466c-a0a5-61007d6c4506/exporter/0.log"
Jul 28 15:22:02.374596 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:02.374557    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zsnpr_103f1358-62b6-466c-a0a5-61007d6c4506/extractor/0.log"
Jul 28 15:22:04.817277 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:04.817246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-56kcz"
Jul 28 15:22:04.864872 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:04.864840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-77bfd6665c-f59p8_c149fbb1-76ac-45c5-8e69-5f7ac4e942d3/manager/0.log"
Jul 28 15:22:05.116418 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:05.116334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-ttblg_4ea6be87-b5f4-491b-bb79-1322bbed538b/manager/0.log"
Jul 28 15:22:05.135665 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:05.135634    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-ttblg_4ea6be87-b5f4-491b-bb79-1322bbed538b/copy-manifests/0.log"
Jul 28 15:22:06.302980 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:06.302947    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cf745d86f-br6wn_8dbc63c8-a969-43af-a377-4fe26198da91/manager/0.log"
Jul 28 15:22:07.265337 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:07.265300    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-k8m7s_d3f0aed6-a460-4914-a4f0-f87dbf300ba1/kube-auth-proxy/0.log"
Jul 28 15:22:07.376832 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:07.376775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:22:07.396261 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:07.396237    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757957ffc-frc2z_3300c232-216d-424b-9ce2-7220f6dbf8a3/router/0.log"
Jul 28 15:22:08.986151 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:08.986118    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-k8m7s_d3f0aed6-a460-4914-a4f0-f87dbf300ba1/kube-auth-proxy/0.log"
Jul 28 15:22:09.081123 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:09.081089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8r4v2_a08dbbf8-7405-44ec-b615-725d3a87abb5/payload-processing/0.log"
Jul 28 15:22:09.095674 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:09.095645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-757957ffc-frc2z_3300c232-216d-424b-9ce2-7220f6dbf8a3/router/0.log"
Jul 28 15:22:11.695753 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:11.695722    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-v5w5j_1e63d39f-f71a-4fdd-a996-ceb64b18ba68/migrator/0.log"
Jul 28 15:22:11.714944 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:11.714915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-v5w5j_1e63d39f-f71a-4fdd-a996-ceb64b18ba68/graceful-termination/0.log"
Jul 28 15:22:13.154912 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.154887    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/kube-multus-additional-cni-plugins/0.log"
Jul 28 15:22:13.173114 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.173086    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/egress-router-binary-copy/0.log"
Jul 28 15:22:13.198268 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.198236    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/cni-plugins/0.log"
Jul 28 15:22:13.217844 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.217820    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/bond-cni-plugin/0.log"
Jul 28 15:22:13.239029 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.238999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/routeoverride-cni/0.log"
Jul 28 15:22:13.257374 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.257345    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/whereabouts-cni-bincopy/0.log"
Jul 28 15:22:13.276005 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.275983    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5fktm_99b6f0b0-9666-4042-ab4a-1887720c36ef/whereabouts-cni/0.log"
Jul 28 15:22:13.634627 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.634594    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zq4qp_cc6329f8-e3de-4d56-8353-c1be192f862c/kube-multus/0.log"
Jul 28 15:22:13.733397 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.733369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sqcgv_b31b90ef-b78e-40a5-89c7-b36dabbc993d/network-metrics-daemon/0.log"
Jul 28 15:22:13.748055 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:13.748030    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sqcgv_b31b90ef-b78e-40a5-89c7-b36dabbc993d/kube-rbac-proxy/0.log"
Jul 28 15:22:14.535063 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.535033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-controller/0.log"
Jul 28 15:22:14.549418 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.549393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/1.log"
Jul 28 15:22:14.549588 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.549515    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovn-acl-logging/0.log"
Jul 28 15:22:14.566841 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.566814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/kube-rbac-proxy-node/0.log"
Jul 28 15:22:14.586799 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.586773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 15:22:14.602901 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.602878    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/northd/0.log"
Jul 28 15:22:14.623331 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.623305    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/nbdb/0.log"
Jul 28 15:22:14.642893 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.642638    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/sbdb/0.log"
Jul 28 15:22:14.714526 ip-10-0-142-230 kubenswrapper[2577]: I0728 15:22:14.714493    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dgs4_0c826ad3-a5c4-47ee-9412-c87ebb211883/ovnkube-controller/0.log"