Jul 17 15:07:03.541756 ip-10-0-128-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 17 15:07:03.541772 ip-10-0-128-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 17 15:07:03.541782 ip-10-0-128-207 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 17 15:07:03.542138 ip-10-0-128-207 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 17 15:07:13.651032 ip-10-0-128-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 17 15:07:13.651051 ip-10-0-128-207 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 604ddc6bee0846b69ad288515cc67c82 --
Jul 17 15:09:09.613394 ip-10-0-128-207 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 15:09:10.002757 ip-10-0-128-207 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 15:09:10.002757 ip-10-0-128-207 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 15:09:10.002757 ip-10-0-128-207 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 15:09:10.002757 ip-10-0-128-207 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 15:09:10.002757 ip-10-0-128-207 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 15:09:10.003505 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.003411    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011551    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011578    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011582    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011585    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011589    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011592    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011595    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.011584 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011598    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011601    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011604    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011606    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011609    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011611    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011614    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011616    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011619    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011621    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011624    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011627    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011629    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011631    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011634    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011637    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011640    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011643    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011645    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011647    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.011893 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011650    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011652    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011655    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011657    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011660    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011662    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011665    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011668    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011671    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011673    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011677    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011681    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011684    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011686    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011690    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011693    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011695    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011698    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011701    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011704    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.012376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011706    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011708    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011711    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011714    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011717    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011720    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011723    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011726    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011728    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011731    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011733    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011736    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011738    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011741    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011743    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011746    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011748    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011751    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011753    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011756    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.012872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011758    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011761    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011763    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011766    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011768    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011771    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011778    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011780    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011783    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011786    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011788    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011791    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011796    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011798    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011801    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011804    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011806    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011809    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011813    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.013430 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011816    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011818    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011821    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011825    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011830    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011833    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011835    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011838    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011841    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011844    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011848    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.011851    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012615    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012623    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012627    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012630    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012633    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012636    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012642    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.013907 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012645    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012648    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012651    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012653    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012657    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012660    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012663    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012666    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012669    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012672    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012675    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012678    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012680    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012683    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012686    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012689    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012692    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012694    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012696    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012699    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.014392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012701    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012704    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012706    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012709    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012712    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012714    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012717    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012720    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012723    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012726    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012728    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012733    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012736    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012738    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012740    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012743    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012745    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012748    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012750    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012753    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.014880 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012755    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012757    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012760    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012764    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012767    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012770    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012772    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012774    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012777    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012779    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012782    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012786    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012789    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012792    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012795    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012799    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012803    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012806    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012808    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.015392 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012811    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012814    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012817    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012821    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012825    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012827    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012830    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012832    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012835    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012838    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012840    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012842    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012845    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012847    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012850    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012852    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012855    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012859    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012861    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.015854 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012864    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012867    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012869    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012872    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012874    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012877    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012880    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012882    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012885    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012888    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012890    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012893    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012895    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.012898    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013373    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013380    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013383    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013388    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013391    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013394    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.016350 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013397    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013400    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013403    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013405    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013408    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013410    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013413    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013416    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013418    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013421    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013424    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013427    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013429    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013432    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013434    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013437    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013440    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013442    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013445    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013448    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013450    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.016844 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013453    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013455    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013457    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013460    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013464    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013467    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013469    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013472    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013475    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013478    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013480    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013483    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013485    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013488    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013490    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013493    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013495    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013498    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013500    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013503    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.017381 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013505    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013508    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013511    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013514    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013516    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013518    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013521    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013525    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013528    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013531    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013534    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013537    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013539    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013542    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013545    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013547    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013551    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013553    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013556    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.017912 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013558    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013561    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013564    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013566    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013569    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013571    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013574    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013576    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013578    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013581    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013583    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013586    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013588    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013590    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013594    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013598    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013602    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013604    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013607    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.018414 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013610    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013613    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013615    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013618    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013620    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013623    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013625    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013628    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013630    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013632    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013638    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013643    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013646    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.013649    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013728    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013736    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013745    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013752    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013759    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013764    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 15:09:10.018864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013771    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013777    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013780    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013783    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013787    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013790    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013793    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013796    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013799    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013802    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013809    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013812    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013816    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013819    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013822    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013825    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013828    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013832    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013835    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013838    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013841    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013844    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013848    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013851    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 15:09:10.019362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013855    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013858    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013862    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013865    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013868    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013871    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013874    2566 flags.go:64] FLAG: --enable-server="true"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013877    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013881    2566 flags.go:64] FLAG: --event-burst="100"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013884    2566 flags.go:64] FLAG: --event-qps="50"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013887    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013891    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013895    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013901    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013905    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013908    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013912    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013915    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013918    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013922    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013925    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013927    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013930    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013933    2566 flags.go:64] FLAG: --feature-gates=""
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013937    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 15:09:10.019932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013940    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013944    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013948    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013951    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013954    2566 flags.go:64] FLAG: --help="false"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013958    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013961    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013964    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013967    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013970    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013973    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013976    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013979    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013982    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013985    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013988    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013991    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013994    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013997    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.013999    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014002    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014005    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014008    2566 flags.go:64] FLAG: --lock-file=""
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014010    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 15:09:10.020627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014013    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014016    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014026    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014031    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014034    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014036    2566 flags.go:64] FLAG: --logging-format="text"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014039    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014043    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014045    2566 flags.go:64] FLAG: --manifest-url=""
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014049    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014066    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014070    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014074    2566 flags.go:64] FLAG: --max-pods="110"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014079    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014082    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014085    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014089    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014092    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014095    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014098    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014105    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014108    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014111    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 15:09:10.021233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014114    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014117    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014131    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014135    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014138    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014141    2566 flags.go:64] FLAG: --port="10250"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014144    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014147    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00ef34f6aa334a3d3"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014150    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014154    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014157    2566 flags.go:64] FLAG: --register-node="true"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014160    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014163    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014167    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014170    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014173    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014177    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014180    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014184    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014187    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014190    2566 flags.go:64] FLAG: --runonce="false"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014192    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014195    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014200    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014203    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 15:09:10.021826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014206    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014209    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014212    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014215    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014218    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014221    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014224    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014227    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014230    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014233    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014235    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014241    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014243    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014246    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014251    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014253    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014256    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014259    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014262    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014265    2566 flags.go:64] FLAG: --v="2"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014269    2566 flags.go:64] FLAG: --version="false"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014273    2566 flags.go:64] FLAG: --vmodule=""
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014277    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.014280    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014404    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.022557 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014410    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014414    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014417    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014420    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014423    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014427    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014430    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014433    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014435    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014438    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014441    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014444    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014447    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014449    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014452    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014455    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014457    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014460    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014462    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014464    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.023206 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014467    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014470    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014472    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014475    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014477    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014480    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014482    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014485    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014488    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014491    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014494    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014496    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014500    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014502    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014505    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014507    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014510    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014514    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014517    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.023696 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014519    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014522    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014524    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014526    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014529    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014531    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014534    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014536    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014540    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014544    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014547    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014549    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014552    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014555    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014558    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014561    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014563    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014566    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014568    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014571    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.024175 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014574    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014576    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014579    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014582    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014585    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014588    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014591    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014593    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014596    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014599    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014605    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014608    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014611    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014614    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014616    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014619    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014621    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014624    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014627    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014629    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.024663 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014632    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014635    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014638    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014642    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014645    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014648    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014651    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014653    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014656    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014658    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014661    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014663    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014666    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014668    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014671    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014674    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014677    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014681    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.025226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.014684    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.015987    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.024588    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.024607    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.024629    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.024636    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024711    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024716    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024719    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024722    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024725    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024728    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024731    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024734    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024737    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.025669 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024739    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024742    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024745    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024747    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024750    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024753    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024756    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024758    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024760    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024763    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024765    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024768    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024770    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024772    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024775    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024778    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024782    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024785    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024788    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.026044 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024791    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024795    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024798    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024800    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024803    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024806    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024808    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024811    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024813    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024816    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024818    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024821    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024823    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024826    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024828    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024831    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024833    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024836    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024838    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024841    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.026540 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024843    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024846    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024850    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024854    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024857    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024859    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024862    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024866    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024869    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024873    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024876    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024879    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024881    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024885    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024888    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024891    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024893    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024896    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024898    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.027015 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024901    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024904    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024906    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024909    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024911    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024914    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024916    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024919    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024921    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024924    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024926    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024929    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024932    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024934    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024937    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024939    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024941    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024944    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024946    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024949    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.027490 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024951    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024953    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024956    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024959    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024961    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024963    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024967    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024970    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024973    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024975    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024978    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.024980    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.024986    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025121    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025128    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 15:09:10.027972 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025131    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025134    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025137    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025139    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025142    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025145    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025147    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025149    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025152    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025154    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025157    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025159    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025162    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025164    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025167    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025169    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025172    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025176    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025178    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025180    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 15:09:10.028357 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025183    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025185    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025188    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025191    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025194    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025197    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025200    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025203    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025205    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025208    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025212    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025216    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025219    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025221    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025224    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025227    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025230    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025233    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025235    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 15:09:10.028864 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025238    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025241    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025243    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025246    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025248    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025251    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025253    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025256    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025258    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025261    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025263    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025266    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025268    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025271    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025273    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025276    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025279    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025282    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025284    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 15:09:10.029402 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025287    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025289    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025292    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025294    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025297    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025299    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025302    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025306    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025309    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025311    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025314    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025316    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025319    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025321    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025323    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025326    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025328    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025330    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025333    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 15:09:10.029853 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025336    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025339    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025342    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025344    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025347    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025349    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025352    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025355    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025357    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025360    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025363    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025365    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025368    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025370    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025372    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025375    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025378    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025380    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025383    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 15:09:10.030338 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:10.025385    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.025390    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.025519    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.027385    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.028667    2566 server.go:1019] "Starting client certificate rotation"
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.028779    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 15:09:10.030816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.028814    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 15:09:10.030997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.030889    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 15:09:10.036126 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.036087    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 15:09:10.046538 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.046518    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 17 15:09:10.048370 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.048354    2566 log.go:25] "Validated CRI v1 image API"
Jul 17 15:09:10.049369 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.049349    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 15:09:10.055869 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.055848    2566 fs.go:135] Filesystem UUIDs: map[52301328-4e40-41f5-bb1d-8b0cff4a04ff:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8166f6f5-6b0b-4d76-826d-45b44d17d090:/dev/nvme0n1p4]
Jul 17 15:09:10.055943 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.055868    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 15:09:10.057623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.057607    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 15:09:10.060592 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.060482    2566 manager.go:216] Machine: {Timestamp:2026-07-17 15:09:10.05994412 +0000 UTC m=+0.335092432 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105591 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223a06098271732d6aa9ab0ced3ee6 SystemUUID:ec223a06-0982-7173-2d6a-a9ab0ced3ee6 BootID:604ddc6b-ee08-46b6-9ad2-88515cc67c82 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:43:64:84:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:43:64:84:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:a0:70:a1:a9:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 15:09:10.060592 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.060590    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 15:09:10.060685 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.060668    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 15:09:10.061956 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.061929    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 15:09:10.062112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.061958    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 15:09:10.062773 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.062763    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 15:09:10.062807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.062775    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 15:09:10.062807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.062783    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 15:09:10.063512 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.063502    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 15:09:10.065117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.065107    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 15:09:10.068118 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068107    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 15:09:10.068250 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068241    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 15:09:10.068284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068272    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 15:09:10.068284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068282    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 15:09:10.068337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068296    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 15:09:10.068337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068308    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 17 15:09:10.068337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068317    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 15:09:10.068766 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068735    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 15:09:10.068891 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.068773    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 15:09:10.071841 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.071819    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 15:09:10.074382 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.074366    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 15:09:10.074449 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.074397    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 15:09:10.075502 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075490    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075508    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075514    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075520    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075530    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075536    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075544    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 15:09:10.075550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075550    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075556    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075562    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075579    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075585    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075609    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 15:09:10.075789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.075614    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 15:09:10.079295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.079275    2566 server.go:1268] "Started kubelet"
Jul 17 15:09:10.080876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.080843    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 15:09:10.080960 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.080910    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 15:09:10.081588 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.081557    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 15:09:10.081851 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.081822    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 15:09:10.081933 ip-10-0-128-207 systemd[1]: Started Kubernetes Kubelet.
Jul 17 15:09:10.082195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.082104    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 15:09:10.082195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.082185    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 15:09:10.083745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.083719    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 15:09:10.083839 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.083822    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 15:09:10.089769 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.089734    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 15:09:10.090494 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.090474    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 15:09:10.090603 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.090491    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 15:09:10.090603 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.090541    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.091009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.090982    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 15:09:10.091159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091126    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 15:09:10.091265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091254    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 15:09:10.091433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091415    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 15:09:10.091433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091430    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 15:09:10.091584 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091455    2566 factory.go:55] Registering systemd factory
Jul 17 15:09:10.091584 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091470    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 17 15:09:10.091709 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091697    2566 factory.go:153] Registering CRI-O factory
Jul 17 15:09:10.091745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091711    2566 factory.go:223] Registration of the crio container factory successfully
Jul 17 15:09:10.091777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091765    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 15:09:10.091816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091790    2566 factory.go:103] Registering Raw factory
Jul 17 15:09:10.091816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.091808    2566 manager.go:1196] Started watching for new ooms in manager
Jul 17 15:09:10.092305 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.091190    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-207.ec2.internal.18c31c42addda23a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-207.ec2.internal,UID:ip-10-0-128-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-207.ec2.internal,},FirstTimestamp:2026-07-17 15:09:10.079251002 +0000 UTC m=+0.354399313,LastTimestamp:2026-07-17 15:09:10.079251002 +0000 UTC m=+0.354399313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-207.ec2.internal,}"
Jul 17 15:09:10.092773 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.092756    2566 manager.go:349] Starting recovery of all containers
Jul 17 15:09:10.094769 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.094744    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sv4j7"
Jul 17 15:09:10.102012 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.101831    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 15:09:10.102191 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.102011    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 15:09:10.102533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.102512    2566 manager.go:354] Recovery completed
Jul 17 15:09:10.103599 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.103580    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sv4j7"
Jul 17 15:09:10.108030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.108014    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.110271 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.110250    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.110360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.110287    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.110360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.110303    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.111158 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.111087    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 15:09:10.111158 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.111102    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 15:09:10.111158 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.111151    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 15:09:10.111325 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.111268    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.112299 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112280    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.112360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112313    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.112360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112329    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.112520 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112505    2566 policy_none.go:49] "None policy: Start"
Jul 17 15:09:10.112549 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112532    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 15:09:10.112549 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.112543    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 15:09:10.112916 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.112842    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-207.ec2.internal.18c31c42afb6f867  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-207.ec2.internal,UID:ip-10-0-128-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-207.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-207.ec2.internal,},FirstTimestamp:2026-07-17 15:09:10.110271591 +0000 UTC m=+0.385419908,LastTimestamp:2026-07-17 15:09:10.110271591 +0000 UTC m=+0.385419908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-207.ec2.internal,}"
Jul 17 15:09:10.113623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.113612    2566 policy_none.go:47] "Start"
Jul 17 15:09:10.150763 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.150743    2566 manager.go:337] "Starting Device Plugin manager"
Jul 17 15:09:10.150907 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.150786    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 15:09:10.150907 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.150799    2566 server.go:87] "Starting device plugin registration server"
Jul 17 15:09:10.151099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.151086    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 15:09:10.151158 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.151100    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 15:09:10.151211 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.151189    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 15:09:10.151310 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.151292    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 15:09:10.151310 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.151307    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 15:09:10.151921 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.151829    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 15:09:10.151921 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.151893    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.233141 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.233103    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 15:09:10.234441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.234418    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 15:09:10.234505 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.234472    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 15:09:10.234542 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.234507    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 15:09:10.234587 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.234552    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 15:09:10.237471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.237448    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:10.251906 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.251889    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.252943 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.252895    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.252943 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.252925    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.252943 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.252936    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.253117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.252960    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.261455 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.261435    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.261511 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.261465    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-207.ec2.internal\": node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.282873 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.282846    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.335094 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.335038    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"]
Jul 17 15:09:10.335254 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.335156    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.336889 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.336873    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.336965 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.336902    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.336965 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.336917    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.338488 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.338472    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.338637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.338621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.338697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.338657    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.339207 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339189    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.339305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339217    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.339305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339231    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.339305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339240    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.339305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339256    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.339305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.339270    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.340543 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.340521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.340620 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.340558    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 15:09:10.341244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.341225    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 15:09:10.341338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.341251    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 15:09:10.341338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.341261    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeHasSufficientPID"
Jul 17 15:09:10.370804 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.370782    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-207.ec2.internal\" not found" node="ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.375367 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.375345    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-207.ec2.internal\" not found" node="ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.383022 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.383005    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.393360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.393332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.393467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.393369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.393467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.393394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2da7476cfcbe41002bd2b6f5093090eb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-207.ec2.internal\" (UID: \"2da7476cfcbe41002bd2b6f5093090eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.483439 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.483397    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.493753 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.493856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.493856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2da7476cfcbe41002bd2b6f5093090eb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-207.ec2.internal\" (UID: \"2da7476cfcbe41002bd2b6f5093090eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.493856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.493856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe42f824519f88ac14289b9352fa2aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal\" (UID: \"fe42f824519f88ac14289b9352fa2aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.493991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.493868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2da7476cfcbe41002bd2b6f5093090eb-config\") pod \"kube-apiserver-proxy-ip-10-0-128-207.ec2.internal\" (UID: \"2da7476cfcbe41002bd2b6f5093090eb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.584244 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.584166    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.673712 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.673683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.680515 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.680497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:10.684733 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.684708    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.785291 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.785240    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.885856 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.885791    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:10.918909 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:10.918872    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:10.987046 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:10.987009    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:11.028411 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.028376    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 15:09:11.029046 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.028550    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 15:09:11.029046 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.028586    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 15:09:11.088278 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:11.088247    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-207.ec2.internal\" not found"
Jul 17 15:09:11.107463 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.107426    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 15:04:10 +0000 UTC" deadline="2028-01-19 02:15:49.914090063 +0000 UTC"
Jul 17 15:09:11.107463 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.107457    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13211h6m38.806636514s"
Jul 17 15:09:11.111413 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.111387    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:11.162791 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:11.162757    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da7476cfcbe41002bd2b6f5093090eb.slice/crio-c886dcf7783173229540a7861973591d9c0ba90059ed103792fa1d1e1d0e359a WatchSource:0}: Error finding container c886dcf7783173229540a7861973591d9c0ba90059ed103792fa1d1e1d0e359a: Status 404 returned error can't find the container with id c886dcf7783173229540a7861973591d9c0ba90059ed103792fa1d1e1d0e359a
Jul 17 15:09:11.163375 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:11.163357    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe42f824519f88ac14289b9352fa2aa2.slice/crio-742912ba169fbd66870c049979e9e556b33f0729d98f99c8c3543f55fd56e3bd WatchSource:0}: Error finding container 742912ba169fbd66870c049979e9e556b33f0729d98f99c8c3543f55fd56e3bd: Status 404 returned error can't find the container with id 742912ba169fbd66870c049979e9e556b33f0729d98f99c8c3543f55fd56e3bd
Jul 17 15:09:11.169363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.169346    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:09:11.191305 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.191271    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:11.204206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.204171    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 15:09:11.208028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.206585    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal"
Jul 17 15:09:11.218598 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.218571    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 15:09:11.238818 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.238734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal" event={"ID":"2da7476cfcbe41002bd2b6f5093090eb","Type":"ContainerStarted","Data":"c886dcf7783173229540a7861973591d9c0ba90059ed103792fa1d1e1d0e359a"}
Jul 17 15:09:11.239875 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.239854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal" event={"ID":"fe42f824519f88ac14289b9352fa2aa2","Type":"ContainerStarted","Data":"742912ba169fbd66870c049979e9e556b33f0729d98f99c8c3543f55fd56e3bd"}
Jul 17 15:09:11.502445 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:11.502360    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:12.023664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.023631    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:12.069957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.069912    2566 apiserver.go:52] "Watching apiserver"
Jul 17 15:09:12.075929 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.075692    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 15:09:12.076749 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.076713    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-8v2c8","openshift-ovn-kubernetes/ovnkube-node-xxh8w","kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f","openshift-cluster-node-tuning-operator/tuned-9dhxh","openshift-dns/node-resolver-4hlbt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal","openshift-multus/multus-additional-cni-plugins-4g29s","openshift-multus/network-metrics-daemon-rrvds","openshift-network-operator/iptables-alerter-hthkx","kube-system/global-pull-secret-syncer-smxd9","kube-system/konnectivity-agent-f4pvj","openshift-image-registry/node-ca-7tx5z","openshift-multus/multus-hxqzc"]
Jul 17 15:09:12.078378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.078351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.080019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.079835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.080533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.080507    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-smslg\""
Jul 17 15:09:12.081394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.080898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.081394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.080905    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.081394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.080951    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.081394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.080898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 15:09:12.081645 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.081582    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.081876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.081854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 15:09:12.082229 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.082087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnqwq\""
Jul 17 15:09:12.082229 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.082123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.082387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.082287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.083247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qf7qq\""
Jul 17 15:09:12.083247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.083247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083089    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.083247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083116    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 15:09:12.083247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 15:09:12.083839 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.083819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.084396 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.084380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 15:09:12.084500 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.084488    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.084770 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.084752    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbs46\""
Jul 17 15:09:12.085352 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.085325    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.085578 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.085558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.088988 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.088378    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.088988 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.088554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.088988 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.088882    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b7j7g\""
Jul 17 15:09:12.090356 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.090327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.090644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.090618    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.090821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.090800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.090915 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.090895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ftzbw\""
Jul 17 15:09:12.091000 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.090912    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:12.091187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.091168    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 15:09:12.092448 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.092044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.094217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.094195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.095048 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.095025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rsxrx\""
Jul 17 15:09:12.095180 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.095111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 15:09:12.095247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.095036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 15:09:12.097662 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.097639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:12.097783 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.097749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:12.097869 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.097860    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 15:09:12.097924 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.097889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 15:09:12.098654 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098632    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-64lkz\""
Jul 17 15:09:12.098767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 15:09:12.098767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098691    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 15:09:12.098767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 15:09:12.098944 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098695    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 15:09:12.098944 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.098830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.098944 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.098902    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:12.099618 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.099600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.101844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.101824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 15:09:12.101972 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.101954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6shsk\""
Jul 17 15:09:12.102140 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.102123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 15:09:12.103594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-etc-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.103688 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-node-log\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.103747 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-run\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.103747 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-kubelet\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.103747 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-var-lib-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.103880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tkx\" (UniqueName: \"kubernetes.io/projected/aad909eb-13a9-4390-aab9-ea7f1497d31b-kube-api-access-h6tkx\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.103880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-host-slash\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.103880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.103994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-multus-daemon-config\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.103994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-registration-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.103994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.103974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-systemd\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-systemd-units\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.104123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hcm\" (UniqueName: \"kubernetes.io/projected/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-kube-api-access-p2hcm\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.104123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.104123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-netd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.104123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-lib-modules\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02bbe719-e77d-43ec-b9e4-9737732089aa-hosts-file\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-script-lib\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hpw\" (UniqueName: \"kubernetes.io/projected/e9e5939e-d8b3-4860-857d-e7d6b6726d31-kube-api-access-h8hpw\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cnibin\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-os-release\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-cni-binary-copy\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn568\" (UniqueName: \"kubernetes.io/projected/29e61a93-0b05-4518-8daf-1209f2e5078d-kube-api-access-mn568\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-log-socket\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e5939e-d8b3-4860-857d-e7d6b6726d31-host\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9e5939e-d8b3-4860-857d-e7d6b6726d31-serviceca\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-conf\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-host\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-system-cni-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-kubelet\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-cnibin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-k8s-cni-cncf-io\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-multus-certs\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-modprobe-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-sys\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbe719-e77d-43ec-b9e4-9737732089aa-tmp-dir\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.104939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-ovn\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.104980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-env-overrides\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-bin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-multus\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-etc-kubernetes\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105090    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-kubernetes\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-iptables-alerter-script\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-socket-dir-parent\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-hostroot\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-var-lib-kubelet\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-tuned\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-tmp\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jjz\" (UniqueName: \"kubernetes.io/projected/02bbe719-e77d-43ec-b9e4-9737732089aa-kube-api-access-97jjz\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8v8\" (UniqueName: \"kubernetes.io/projected/303737d8-f0e3-4189-9a8f-31a72d3cd29a-kube-api-access-mc8v8\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9cn\" (UniqueName: \"kubernetes.io/projected/11685444-cbce-45a0-8228-5bb70de342fe-kube-api-access-wv9cn\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.105526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovn-node-metrics-cert\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-os-release\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-netns\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-socket-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-sys-fs\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-slash\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-bin\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-device-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-netns\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-conf-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwtp\" (UniqueName: \"kubernetes.io/projected/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kube-api-access-fpwtp\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-systemd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-config\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.106247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-system-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.106914 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysconfig\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.106914 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.105980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvqj\" (UniqueName: \"kubernetes.io/projected/e455437e-2d8c-452a-874d-287c57ebc381-kube-api-access-xgvqj\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.131364 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.131339    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s7jb9"
Jul 17 15:09:12.141704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.141674    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s7jb9"
Jul 17 15:09:12.192512 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.192482    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 15:09:12.206549 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-netns\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.206549 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-socket-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-sys-fs\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-netns\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-slash\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-bin\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.206797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-device-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-socket-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-bin\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-device-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-netns\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-sys-fs\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-conf-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-slash\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwtp\" (UniqueName: \"kubernetes.io/projected/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kube-api-access-fpwtp\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-netns\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-systemd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-conf-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.206988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-config\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-systemd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.207143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-system-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysconfig\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysconfig\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvqj\" (UniqueName: \"kubernetes.io/projected/e455437e-2d8c-452a-874d-287c57ebc381-kube-api-access-xgvqj\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-system-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-etc-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-node-log\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-run\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-kubelet\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207350    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-node-log\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-var-lib-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-run\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-var-lib-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tkx\" (UniqueName: \"kubernetes.io/projected/aad909eb-13a9-4390-aab9-ea7f1497d31b-kube-api-access-h6tkx\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-host-slash\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-kubelet\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.208034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-multus-daemon-config\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-etc-openvswitch\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-host-slash\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-registration-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-systemd\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-systemd-units\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-registration-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-config\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-systemd\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hcm\" (UniqueName: \"kubernetes.io/projected/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-kube-api-access-p2hcm\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-systemd-units\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-kubelet-config\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-netd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-cni-netd\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-lib-modules\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.210340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02bbe719-e77d-43ec-b9e4-9737732089aa-hosts-file\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-script-lib\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-binary-copy\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.207966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hpw\" (UniqueName: \"kubernetes.io/projected/e9e5939e-d8b3-4860-857d-e7d6b6726d31-kube-api-access-h8hpw\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02bbe719-e77d-43ec-b9e4-9737732089aa-hosts-file\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-lib-modules\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-multus-daemon-config\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cnibin\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cnibin\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-os-release\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-cni-binary-copy\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn568\" (UniqueName: \"kubernetes.io/projected/29e61a93-0b05-4518-8daf-1209f2e5078d-kube-api-access-mn568\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-log-socket\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e5939e-d8b3-4860-857d-e7d6b6726d31-host\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208220    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-os-release\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9e5939e-d8b3-4860-857d-e7d6b6726d31-serviceca\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.211053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-log-socket\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-conf\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-host\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9e5939e-d8b3-4860-857d-e7d6b6726d31-host\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-system-cni-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.208408    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-host\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-sysctl-conf\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-system-cni-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-kubelet\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.208513    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:12.708470221 +0000 UTC m=+2.983618540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-kubelet\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.211844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11685444-cbce-45a0-8228-5bb70de342fe-cni-binary-copy\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovnkube-script-lib\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-cnibin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-k8s-cni-cncf-io\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-cni-dir\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-multus-certs\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-k8s-cni-cncf-io\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-modprobe-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-cnibin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-run-multus-certs\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-sys\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9e5939e-d8b3-4860-857d-e7d6b6726d31-serviceca\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbe719-e77d-43ec-b9e4-9737732089aa-tmp-dir\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-ovn\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-modprobe-d\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.212619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-sys\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-env-overrides\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-bin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-multus\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-bin\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-etc-kubernetes\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.208935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aad909eb-13a9-4390-aab9-ea7f1497d31b-run-ovn\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-kubernetes\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-etc-kubernetes\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-iptables-alerter-script\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-host-var-lib-cni-multus\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-kubernetes\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-konnectivity-ca\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-dbus\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.213387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-socket-dir-parent\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02bbe719-e77d-43ec-b9e4-9737732089aa-tmp-dir\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aad909eb-13a9-4390-aab9-ea7f1497d31b-env-overrides\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-hostroot\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-var-lib-kubelet\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-tuned\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-hostroot\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11685444-cbce-45a0-8228-5bb70de342fe-multus-socket-dir-parent\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e61a93-0b05-4518-8daf-1209f2e5078d-var-lib-kubelet\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-tmp\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jjz\" (UniqueName: \"kubernetes.io/projected/02bbe719-e77d-43ec-b9e4-9737732089aa-kube-api-access-97jjz\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/303737d8-f0e3-4189-9a8f-31a72d3cd29a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8v8\" (UniqueName: \"kubernetes.io/projected/303737d8-f0e3-4189-9a8f-31a72d3cd29a-kube-api-access-mc8v8\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9cn\" (UniqueName: \"kubernetes.io/projected/11685444-cbce-45a0-8228-5bb70de342fe-kube-api-access-wv9cn\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovn-node-metrics-cert\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-os-release\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-agent-certs\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.214024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209654    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 15:09:12.214743 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-iptables-alerter-script\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.214743 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.209726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/303737d8-f0e3-4189-9a8f-31a72d3cd29a-os-release\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.215418 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.215379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-tmp\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.215532 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.215434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29e61a93-0b05-4518-8daf-1209f2e5078d-etc-tuned\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.217299 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.217272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aad909eb-13a9-4390-aab9-ea7f1497d31b-ovn-node-metrics-cert\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.218856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.218837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwtp\" (UniqueName: \"kubernetes.io/projected/99ca0b3d-a777-4c56-85a2-53a7a8c83bd4-kube-api-access-fpwtp\") pod \"aws-ebs-csi-driver-node-jw24f\" (UID: \"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.222854 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.222832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tkx\" (UniqueName: \"kubernetes.io/projected/aad909eb-13a9-4390-aab9-ea7f1497d31b-kube-api-access-h6tkx\") pod \"ovnkube-node-xxh8w\" (UID: \"aad909eb-13a9-4390-aab9-ea7f1497d31b\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.222956 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.222934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvqj\" (UniqueName: \"kubernetes.io/projected/e455437e-2d8c-452a-874d-287c57ebc381-kube-api-access-xgvqj\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.223296 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.223274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hcm\" (UniqueName: \"kubernetes.io/projected/b297e9fb-e6ce-4c83-8de8-93a2c13369aa-kube-api-access-p2hcm\") pod \"iptables-alerter-hthkx\" (UID: \"b297e9fb-e6ce-4c83-8de8-93a2c13369aa\") " pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.223763 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.223721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hpw\" (UniqueName: \"kubernetes.io/projected/e9e5939e-d8b3-4860-857d-e7d6b6726d31-kube-api-access-h8hpw\") pod \"node-ca-7tx5z\" (UID: \"e9e5939e-d8b3-4860-857d-e7d6b6726d31\") " pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.224976 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.224931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8v8\" (UniqueName: \"kubernetes.io/projected/303737d8-f0e3-4189-9a8f-31a72d3cd29a-kube-api-access-mc8v8\") pod \"multus-additional-cni-plugins-4g29s\" (UID: \"303737d8-f0e3-4189-9a8f-31a72d3cd29a\") " pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.225351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.225329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jjz\" (UniqueName: \"kubernetes.io/projected/02bbe719-e77d-43ec-b9e4-9737732089aa-kube-api-access-97jjz\") pod \"node-resolver-4hlbt\" (UID: \"02bbe719-e77d-43ec-b9e4-9737732089aa\") " pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.225989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.225971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn568\" (UniqueName: \"kubernetes.io/projected/29e61a93-0b05-4518-8daf-1209f2e5078d-kube-api-access-mn568\") pod \"tuned-9dhxh\" (UID: \"29e61a93-0b05-4518-8daf-1209f2e5078d\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.227953 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.227934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9cn\" (UniqueName: \"kubernetes.io/projected/11685444-cbce-45a0-8228-5bb70de342fe-kube-api-access-wv9cn\") pod \"multus-hxqzc\" (UID: \"11685444-cbce-45a0-8228-5bb70de342fe\") " pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.310989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.310904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-kubelet-config\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.310989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.310979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-konnectivity-ca\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-kubelet-config\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-dbus\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-agent-certs\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.311263 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.311177    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:12.311556 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.311289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:12.811268268 +0000 UTC m=+3.086416583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:12.311611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-dbus\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.311887 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.311844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-konnectivity-ca\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.314212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.314183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a440e36a-8a41-4911-94e9-1c1ddb6e3eaa-agent-certs\") pod \"konnectivity-agent-f4pvj\" (UID: \"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa\") " pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.318321 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.318295    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:12.318321 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.318325    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:12.318475 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.318340    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:12.318475 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.318406    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:12.818387346 +0000 UTC m=+3.093535651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:12.395100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.395040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hthkx"
Jul 17 15:09:12.403572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.403533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dhxh"
Jul 17 15:09:12.412380 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.412350    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7tx5z"
Jul 17 15:09:12.417285 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.417260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxqzc"
Jul 17 15:09:12.423675 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.423638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f"
Jul 17 15:09:12.430315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.430293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hlbt"
Jul 17 15:09:12.436408 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.436389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4g29s"
Jul 17 15:09:12.443160 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.443120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:12.448333 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.448303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:12.714529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.714427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:12.714694 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.714603    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:12.714694 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.714680    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:13.71466252 +0000 UTC m=+3.989810821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:12.815458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.815406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:12.815708 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.815635    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:12.815790 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.815724    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:13.815701 +0000 UTC m=+4.090849315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:12.916853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:12.916815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:12.917034 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.916959    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:12.917034 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.916988    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:12.917034 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.917002    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:12.917180 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:12.917100    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:13.917052615 +0000 UTC m=+4.192200916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:13.143318 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.143283    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ca0b3d_a777_4c56_85a2_53a7a8c83bd4.slice/crio-acf99369670aaf920da720ef469a5bcadb674304722dcdcee5978765448205db WatchSource:0}: Error finding container acf99369670aaf920da720ef469a5bcadb674304722dcdcee5978765448205db: Status 404 returned error can't find the container with id acf99369670aaf920da720ef469a5bcadb674304722dcdcee5978765448205db
Jul 17 15:09:13.144178 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.144113    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 15:04:12 +0000 UTC" deadline="2027-12-15 15:53:58.477608367 +0000 UTC"
Jul 17 15:09:13.144294 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.144180    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12384h44m45.333433649s"
Jul 17 15:09:13.145456 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.145431    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e61a93_0b05_4518_8daf_1209f2e5078d.slice/crio-dab24dfc9f90ad977ec5dd3b207c1ecbb854e29f16ec9a82ee52d6acfec2ccbf WatchSource:0}: Error finding container dab24dfc9f90ad977ec5dd3b207c1ecbb854e29f16ec9a82ee52d6acfec2ccbf: Status 404 returned error can't find the container with id dab24dfc9f90ad977ec5dd3b207c1ecbb854e29f16ec9a82ee52d6acfec2ccbf
Jul 17 15:09:13.148265 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.146487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11685444_cbce_45a0_8228_5bb70de342fe.slice/crio-21c5171441aa1ac06ca81143ef901b51cc67dee6895d31a4c15f891b90a5e7ca WatchSource:0}: Error finding container 21c5171441aa1ac06ca81143ef901b51cc67dee6895d31a4c15f891b90a5e7ca: Status 404 returned error can't find the container with id 21c5171441aa1ac06ca81143ef901b51cc67dee6895d31a4c15f891b90a5e7ca
Jul 17 15:09:13.148265 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.147861    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb297e9fb_e6ce_4c83_8de8_93a2c13369aa.slice/crio-2d1fd43461641e45ae6fda548f1f33bccdcd308cbf2bed9751cb8706869d52a2 WatchSource:0}: Error finding container 2d1fd43461641e45ae6fda548f1f33bccdcd308cbf2bed9751cb8706869d52a2: Status 404 returned error can't find the container with id 2d1fd43461641e45ae6fda548f1f33bccdcd308cbf2bed9751cb8706869d52a2
Jul 17 15:09:13.153226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.150670    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303737d8_f0e3_4189_9a8f_31a72d3cd29a.slice/crio-6a2f3f269cfb0aa2c8018bd7bf1aa62bd52c3593c7f2ae0e6962628a18f3014a WatchSource:0}: Error finding container 6a2f3f269cfb0aa2c8018bd7bf1aa62bd52c3593c7f2ae0e6962628a18f3014a: Status 404 returned error can't find the container with id 6a2f3f269cfb0aa2c8018bd7bf1aa62bd52c3593c7f2ae0e6962628a18f3014a
Jul 17 15:09:13.153226 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.152021    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bbe719_e77d_43ec_b9e4_9737732089aa.slice/crio-c847aa6e7e2e03560986d229fac59fc162ee4506602b076f5faf9cebbf831a53 WatchSource:0}: Error finding container c847aa6e7e2e03560986d229fac59fc162ee4506602b076f5faf9cebbf831a53: Status 404 returned error can't find the container with id c847aa6e7e2e03560986d229fac59fc162ee4506602b076f5faf9cebbf831a53
Jul 17 15:09:13.153607 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.153554    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda440e36a_8a41_4911_94e9_1c1ddb6e3eaa.slice/crio-20a280b2660d8bd3a3795a009bbdb1cab4c6f55fd46d0ca19649d41bea145670 WatchSource:0}: Error finding container 20a280b2660d8bd3a3795a009bbdb1cab4c6f55fd46d0ca19649d41bea145670: Status 404 returned error can't find the container with id 20a280b2660d8bd3a3795a009bbdb1cab4c6f55fd46d0ca19649d41bea145670
Jul 17 15:09:13.154892 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.154441    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad909eb_13a9_4390_aab9_ea7f1497d31b.slice/crio-e4288fef77646f9b383b27ef249de0fd677d82f84d6506aff97cd68a48173a1b WatchSource:0}: Error finding container e4288fef77646f9b383b27ef249de0fd677d82f84d6506aff97cd68a48173a1b: Status 404 returned error can't find the container with id e4288fef77646f9b383b27ef249de0fd677d82f84d6506aff97cd68a48173a1b
Jul 17 15:09:13.156376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:13.156343    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e5939e_d8b3_4860_857d_e7d6b6726d31.slice/crio-375f0aa5699abbe7ace62adc8838a2769fa0f8ff686dcd7dac9e30c162328931 WatchSource:0}: Error finding container 375f0aa5699abbe7ace62adc8838a2769fa0f8ff686dcd7dac9e30c162328931: Status 404 returned error can't find the container with id 375f0aa5699abbe7ace62adc8838a2769fa0f8ff686dcd7dac9e30c162328931
Jul 17 15:09:13.234826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.234724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:13.234910 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.234860    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:13.247418 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.247377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f" event={"ID":"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4","Type":"ContainerStarted","Data":"acf99369670aaf920da720ef469a5bcadb674304722dcdcee5978765448205db"}
Jul 17 15:09:13.250627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.250593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"e4288fef77646f9b383b27ef249de0fd677d82f84d6506aff97cd68a48173a1b"}
Jul 17 15:09:13.255731 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.255693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4pvj" event={"ID":"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa","Type":"ContainerStarted","Data":"20a280b2660d8bd3a3795a009bbdb1cab4c6f55fd46d0ca19649d41bea145670"}
Jul 17 15:09:13.258667 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.258573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hlbt" event={"ID":"02bbe719-e77d-43ec-b9e4-9737732089aa","Type":"ContainerStarted","Data":"c847aa6e7e2e03560986d229fac59fc162ee4506602b076f5faf9cebbf831a53"}
Jul 17 15:09:13.262273 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.262239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hthkx" event={"ID":"b297e9fb-e6ce-4c83-8de8-93a2c13369aa","Type":"ContainerStarted","Data":"2d1fd43461641e45ae6fda548f1f33bccdcd308cbf2bed9751cb8706869d52a2"}
Jul 17 15:09:13.263579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.263522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxqzc" event={"ID":"11685444-cbce-45a0-8228-5bb70de342fe","Type":"ContainerStarted","Data":"21c5171441aa1ac06ca81143ef901b51cc67dee6895d31a4c15f891b90a5e7ca"}
Jul 17 15:09:13.264849 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.264826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dhxh" event={"ID":"29e61a93-0b05-4518-8daf-1209f2e5078d","Type":"ContainerStarted","Data":"dab24dfc9f90ad977ec5dd3b207c1ecbb854e29f16ec9a82ee52d6acfec2ccbf"}
Jul 17 15:09:13.276133 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.276094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal" event={"ID":"2da7476cfcbe41002bd2b6f5093090eb","Type":"ContainerStarted","Data":"e2dad6a2235da9f744fb1b9fa83b649d01e6efd2556bb7f289524ae377e23b70"}
Jul 17 15:09:13.277797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.277768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tx5z" event={"ID":"e9e5939e-d8b3-4860-857d-e7d6b6726d31","Type":"ContainerStarted","Data":"375f0aa5699abbe7ace62adc8838a2769fa0f8ff686dcd7dac9e30c162328931"}
Jul 17 15:09:13.281929 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.281898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerStarted","Data":"6a2f3f269cfb0aa2c8018bd7bf1aa62bd52c3593c7f2ae0e6962628a18f3014a"}
Jul 17 15:09:13.298614 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.298549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-207.ec2.internal" podStartSLOduration=2.298524991 podStartE2EDuration="2.298524991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:13.298045575 +0000 UTC m=+3.573193896" watchObservedRunningTime="2026-07-17 15:09:13.298524991 +0000 UTC m=+3.573673317"
Jul 17 15:09:13.724176 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.723778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:13.724176 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.724008    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:13.724176 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.724088    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:15.724068275 +0000 UTC m=+5.999216582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:13.825160 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.825107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:13.825391 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.825316    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:13.825391 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.825385    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:15.82536601 +0000 UTC m=+6.100514329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:13.925904 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:13.925861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:13.926167 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.926118    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:13.926167 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.926143    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:13.926167 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.926159    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:13.926356 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:13.926227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:15.926207415 +0000 UTC m=+6.201355737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:14.107026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.106607    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 15:09:14.145173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.145081    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 15:04:12 +0000 UTC" deadline="2028-04-16 08:11:42.279314604 +0000 UTC"
Jul 17 15:09:14.145173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.145147    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h2m28.134174072s"
Jul 17 15:09:14.236873 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.236275    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:14.236873 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:14.236449    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:14.236873 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.236502    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:14.236873 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:14.236638    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:14.318486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.318439    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe42f824519f88ac14289b9352fa2aa2" containerID="bfa075eb82f7f2833eb9472181d574c29b87cb8a68aecdea6a8ad4eec993f12e" exitCode=0
Jul 17 15:09:14.319041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:14.319004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal" event={"ID":"fe42f824519f88ac14289b9352fa2aa2","Type":"ContainerDied","Data":"bfa075eb82f7f2833eb9472181d574c29b87cb8a68aecdea6a8ad4eec993f12e"}
Jul 17 15:09:15.235025 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.234985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:15.235589 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.235150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:15.338104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.337400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal" event={"ID":"fe42f824519f88ac14289b9352fa2aa2","Type":"ContainerStarted","Data":"38ccc9000cc89221a578812779cd11ea58d237870ee4a55e05462d232cb01c5d"}
Jul 17 15:09:15.358318 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.357011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-207.ec2.internal" podStartSLOduration=4.356986122 podStartE2EDuration="4.356986122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:15.356746784 +0000 UTC m=+5.631895193" watchObservedRunningTime="2026-07-17 15:09:15.356986122 +0000 UTC m=+5.632134447"
Jul 17 15:09:15.742944 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.742234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:15.742944 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.742453    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:15.742944 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.742524    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:19.742502918 +0000 UTC m=+10.017651219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:15.843296 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.843253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:15.843480 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.843420    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:15.843558 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.843491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:19.843470345 +0000 UTC m=+10.118618652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:15.944714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:15.943951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:15.944714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.944159    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:15.944714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.944186    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:15.944714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.944202    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:15.944714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:15.944271    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:19.944248515 +0000 UTC m=+10.219396833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:16.235837 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:16.235733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:16.235837 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:16.235759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:16.236437 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:16.235892    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:16.236437 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:16.236048    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:17.235111 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:17.235058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:17.235331 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:17.235238    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:18.237674 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:18.237641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:18.238151 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:18.237758    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:18.238151 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:18.238069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:18.238246 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:18.238156    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:19.234985 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:19.234875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:19.235250 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.235009    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:19.781735 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:19.781700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:19.782183 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.782035    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:19.782183 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.782102    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:27.782086528 +0000 UTC m=+18.057234827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:19.882875 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:19.882833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:19.883107 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.883031    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:19.883174 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.883120    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:27.883095742 +0000 UTC m=+18.158244055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:19.984254 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:19.984211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:19.984431 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.984418    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:19.984487 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.984442    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:19.984487 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.984456    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:19.984608 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:19.984522    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:27.984504277 +0000 UTC m=+18.259652599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:20.236149 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:20.236046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:20.236300 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:20.236189    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:20.236300 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:20.236228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:20.236424 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:20.236339    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:21.234728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:21.234689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:21.235174 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:21.234848    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:22.236750 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:22.236168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:22.236750 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:22.236313    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:22.237536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:22.237324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:22.237536 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:22.237448    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:23.235919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:23.235474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:23.235919 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:23.235616    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:24.234955 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:24.234919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:24.235417 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:24.234928    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:24.235417 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:24.235080    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:24.235417 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:24.235135    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:25.235707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:25.235668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:25.236213 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:25.235802    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:26.235418 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:26.235373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:26.235600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:26.235384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:26.235600 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:26.235531    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:26.235600 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:26.235583    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:27.235582 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:27.235545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:27.236010 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:27.235677    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:27.843194 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:27.843154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:27.843397 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:27.843316    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:27.843462 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:27.843401    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:43.843371297 +0000 UTC m=+34.118519599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:27.943690 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:27.943653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:27.943850 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:27.943813    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:27.943924 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:27.943884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:43.943864403 +0000 UTC m=+34.219012719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:28.044238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:28.044208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:28.044396 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.044354    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:28.044396 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.044372    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:28.044396 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.044381    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:28.044567 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.044430    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:44.044415356 +0000 UTC m=+34.319563656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:28.235413 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:28.235306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:28.235413 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:28.235306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:28.235629 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.235470    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:28.235629 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:28.235512    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:29.235086 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:29.235031    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:29.235282 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:29.235169    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:30.236394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:30.236342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:30.236881 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:30.236448    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:30.236881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:30.236490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:30.236881 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:30.236615    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:31.235353 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:31.235315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:31.235557 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:31.235445    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:32.235773 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:32.235742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:32.236266 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:32.235749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:32.236266 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:32.235854    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:32.236266 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:32.235928    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:33.235974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.235621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:33.236717 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:33.236093    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:33.375283 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:09:33.375703 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375673    2566 generic.go:356] "Generic (PLEG): container finished" podID="aad909eb-13a9-4390-aab9-ea7f1497d31b" containerID="b3f037a9b30e3a93e6279153333b89e677ab5ef921628e80d49caab8597a62f3" exitCode=1
Jul 17 15:09:33.375817 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"295bdc3dbca155ba76ef251733a85a06c274be5884216d691d8bcb33f6e8f5a1"}
Jul 17 15:09:33.375862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"a4f9f94c9f4989188d0b5873d70b4840704909cd5903133cc252003706a2d7a2"}
Jul 17 15:09:33.375903 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"d1d29d8964cb1dc8c6527bd95b66a18622a2f5986841493dc0a99e75d441603e"}
Jul 17 15:09:33.375950 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"01f0d4a7bf87d54d0c612b8dbc3bb9ea623962db15c429799ca63d6bd80f6155"}
Jul 17 15:09:33.375950 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerDied","Data":"b3f037a9b30e3a93e6279153333b89e677ab5ef921628e80d49caab8597a62f3"}
Jul 17 15:09:33.376020 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.375970    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"9cbda1582c88317fb53d5e171b6be8ca0de229bd9580a39035e77d60f3cc6f31"}
Jul 17 15:09:33.377436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.377402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4pvj" event={"ID":"a440e36a-8a41-4911-94e9-1c1ddb6e3eaa","Type":"ContainerStarted","Data":"52939ee20f5b7f7edeac44eef838373b88030ff0564c6f4359042a2c627c62fd"}
Jul 17 15:09:33.379360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.379334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hlbt" event={"ID":"02bbe719-e77d-43ec-b9e4-9737732089aa","Type":"ContainerStarted","Data":"c63525d9b64d03e27c32a05fbfd2a2b229e26e34b379d4c882f96ce69c0d6ca4"}
Jul 17 15:09:33.380985 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.380957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxqzc" event={"ID":"11685444-cbce-45a0-8228-5bb70de342fe","Type":"ContainerStarted","Data":"8a1e5aa01f3a819295fe179a73a7bc0ad92e53b6cc13e0defd72dfcd5208d190"}
Jul 17 15:09:33.382607 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.382579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dhxh" event={"ID":"29e61a93-0b05-4518-8daf-1209f2e5078d","Type":"ContainerStarted","Data":"0a0d9ebbf0940f356f9d926a6d3b6eb622ef57401f59645b046421290c48460d"}
Jul 17 15:09:33.384254 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.384233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7tx5z" event={"ID":"e9e5939e-d8b3-4860-857d-e7d6b6726d31","Type":"ContainerStarted","Data":"e417bfbca0c487fb0290ebd56c9297542198eae60ae0b40bfbd8eca1ef46cbca"}
Jul 17 15:09:33.386049 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.386019    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="5c923c819c0209891be6fbb86b6ce37a172126579b908d1b90b3b10d4b004dfa" exitCode=0
Jul 17 15:09:33.386164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.386093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"5c923c819c0209891be6fbb86b6ce37a172126579b908d1b90b3b10d4b004dfa"}
Jul 17 15:09:33.387853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.387782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f" event={"ID":"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4","Type":"ContainerStarted","Data":"c6b31dbf1c8648f3fa9852ad1d6050a9d1bbf9b389cd4d44a4c538d2ee03d390"}
Jul 17 15:09:33.396380 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.396320    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f4pvj" podStartSLOduration=4.112218236 podStartE2EDuration="23.39630412s" totalImagesPullingTime="19.284085884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:33.395450831 +0000 UTC m=+23.670599151" watchObservedRunningTime="2026-07-17 15:09:33.39630412 +0000 UTC m=+23.671452440"
Jul 17 15:09:33.415300 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.415209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hxqzc" podStartSLOduration=3.833570585 podStartE2EDuration="23.415192568s" totalImagesPullingTime="19.581621983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:33.414601388 +0000 UTC m=+23.689749710" watchObservedRunningTime="2026-07-17 15:09:33.415192568 +0000 UTC m=+23.690340890"
Jul 17 15:09:33.431679 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.431623    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4hlbt" podStartSLOduration=4.147551794 podStartE2EDuration="23.431608975s" totalImagesPullingTime="19.284057181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:33.431239454 +0000 UTC m=+23.706387776" watchObservedRunningTime="2026-07-17 15:09:33.431608975 +0000 UTC m=+23.706757292"
Jul 17 15:09:33.552269 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.552183    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7tx5z" podStartSLOduration=4.307420894 podStartE2EDuration="23.552166874s" totalImagesPullingTime="19.24474598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:33.526217739 +0000 UTC m=+23.801366061" watchObservedRunningTime="2026-07-17 15:09:33.552166874 +0000 UTC m=+23.827315227"
Jul 17 15:09:33.552548 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.552513    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9dhxh" podStartSLOduration=4.245316617 podStartE2EDuration="23.552509237s" totalImagesPullingTime="19.30719262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:33.552005072 +0000 UTC m=+23.827153393" watchObservedRunningTime="2026-07-17 15:09:33.552509237 +0000 UTC m=+23.827657563"
Jul 17 15:09:33.562122 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.562089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:33.562770 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.562753    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:33.671336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:33.671316    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 15:09:34.164224 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.164030    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T15:09:33.671331031Z","UUID":"ce8514e8-8ad8-4922-91b9-8b1e04dd93ff","Handler":null,"Name":"","Endpoint":""}
Jul 17 15:09:34.165970 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.165936    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 15:09:34.166130 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.165981    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 15:09:34.237013 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.236984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:34.237446 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:34.237123    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:34.237446 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.237201    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:34.237446 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:34.237293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:34.392185 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.392145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hthkx" event={"ID":"b297e9fb-e6ce-4c83-8de8-93a2c13369aa","Type":"ContainerStarted","Data":"0305ff05145370d9cf471111251d616f021940348d77af6045780ba457eed39c"}
Jul 17 15:09:34.394753 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.394721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f" event={"ID":"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4","Type":"ContainerStarted","Data":"faac3eb7c08cdca22bb301d0d7c9bb2d0a0f69b5b7a784d48e8eb9e085ad563c"}
Jul 17 15:09:34.395601 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.395581    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:34.395725 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.395641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f4pvj"
Jul 17 15:09:34.426576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:34.426293    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hthkx" podStartSLOduration=5.116680891 podStartE2EDuration="24.426271435s" totalImagesPullingTime="19.309590544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:34.408979915 +0000 UTC m=+24.684128238" watchObservedRunningTime="2026-07-17 15:09:34.426271435 +0000 UTC m=+24.701419757"
Jul 17 15:09:35.234944 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:35.234855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:35.235101 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:35.234981    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:35.401479 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:35.401253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f" event={"ID":"99ca0b3d-a777-4c56-85a2-53a7a8c83bd4","Type":"ContainerStarted","Data":"6cf490d1abf5e1b8fc592aaa2160a4545ec2ab782dd97ea4b6e76aa884a8a706"}
Jul 17 15:09:35.420047 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:35.420001    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jw24f" podStartSLOduration=4.047273158 podStartE2EDuration="25.419985116s" totalImagesPullingTime="21.372711958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:35.418930751 +0000 UTC m=+25.694079072" watchObservedRunningTime="2026-07-17 15:09:35.419985116 +0000 UTC m=+25.695133438"
Jul 17 15:09:36.235518 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:36.235473    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:36.235729 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:36.235478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:36.235729 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:36.235618    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:36.235729 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:36.235675    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:36.406912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:36.406882    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:09:36.407894 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:36.407866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"68581593f2c870af629998850d514fb19cd80c99c55e0e25166fbdee58a1df84"}
Jul 17 15:09:37.235173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:37.235137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:37.235362 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:37.235260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:38.235288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.235104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:38.236038 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.235115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:38.236038 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:38.235394    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:38.236038 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:38.235442    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:38.413288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.413198    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="5f1236a88fb537e03d424b2399398f5d5e6b5910be42009d89678b61886b7e13" exitCode=0
Jul 17 15:09:38.413288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.413276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"5f1236a88fb537e03d424b2399398f5d5e6b5910be42009d89678b61886b7e13"}
Jul 17 15:09:38.416671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.416651    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:09:38.417069 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.417030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"e3abffe140e1d44e346c29cb5391b9a0e4e31da79703c7faecf4ee5f33420982"}
Jul 17 15:09:38.417389 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.417370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:38.417480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.417425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:38.417480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.417471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:38.417578 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.417555    2566 scope.go:117] "RemoveContainer" containerID="b3f037a9b30e3a93e6279153333b89e677ab5ef921628e80d49caab8597a62f3"
Jul 17 15:09:38.433644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.433612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:38.434424 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:38.434410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:09:39.234911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.234879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:39.235102 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:39.234983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:39.422088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.421805    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="41a5d404b3e90d7848fb4f3b263f6234ee86d30a8ba79f51cce23d85602fedc2" exitCode=0
Jul 17 15:09:39.422088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.421871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"41a5d404b3e90d7848fb4f3b263f6234ee86d30a8ba79f51cce23d85602fedc2"}
Jul 17 15:09:39.425953 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.425932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:09:39.426323 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.426300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" event={"ID":"aad909eb-13a9-4390-aab9-ea7f1497d31b","Type":"ContainerStarted","Data":"48e47c725de86a8e4cc74f137048e70fe03b039d07b39254767779a61346b590"}
Jul 17 15:09:39.466321 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.466261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w" podStartSLOduration=10.136089611 podStartE2EDuration="29.466241818s" totalImagesPullingTime="19.330152207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:39.466123637 +0000 UTC m=+29.741271957" watchObservedRunningTime="2026-07-17 15:09:39.466241818 +0000 UTC m=+29.741390126"
Jul 17 15:09:39.528969 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.528940    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rrvds"]
Jul 17 15:09:39.529156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.529077    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:39.529226 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:39.529184    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:39.529648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.529630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8v2c8"]
Jul 17 15:09:39.529725 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.529714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:39.529810 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:39.529788    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:39.530511 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.530489    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smxd9"]
Jul 17 15:09:39.530623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:39.530571    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:39.530683 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:39.530667    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:40.431114 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:40.431048    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="955d31231f52e00ccbe268cca1bf83a8b3544b64553c096e388ab2bf39f91fef" exitCode=0
Jul 17 15:09:40.431114 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:40.431098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"955d31231f52e00ccbe268cca1bf83a8b3544b64553c096e388ab2bf39f91fef"}
Jul 17 15:09:41.234963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:41.234879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:41.234963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:41.234904    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:41.234963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:41.234916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:41.235289 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:41.235042    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:41.235289 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:41.235129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:41.235289 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:41.235200    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:43.235764 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:43.235721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:43.236475 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:43.235730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:43.236475 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.235858    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:43.236475 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.235931    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:43.236475 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:43.235743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:43.236475 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.236026    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:43.867080 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:43.867025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:43.867257 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.867181    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:43.867306 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.867267    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs podName:e455437e-2d8c-452a-874d-287c57ebc381 nodeName:}" failed. No retries permitted until 2026-07-17 15:10:15.867244496 +0000 UTC m=+66.142392818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs") pod "network-metrics-daemon-rrvds" (UID: "e455437e-2d8c-452a-874d-287c57ebc381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 15:09:43.967868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:43.967769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:43.968029 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.967945    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:43.968029 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:43.968026    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret podName:da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39 nodeName:}" failed. No retries permitted until 2026-07-17 15:10:15.967993966 +0000 UTC m=+66.243142265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret") pod "global-pull-secret-syncer-smxd9" (UID: "da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39") : object "kube-system"/"original-pull-secret" not registered
Jul 17 15:09:44.068462 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:44.068422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:44.068652 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:44.068625    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 15:09:44.068652 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:44.068652    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 15:09:44.068764 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:44.068665    2566 projected.go:196] Error preparing data for projected volume kube-api-access-htmm8 for pod openshift-network-diagnostics/network-check-target-8v2c8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:44.068764 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:44.068734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8 podName:03ec8278-12ed-4a29-b1cc-d3722d3a3b13 nodeName:}" failed. No retries permitted until 2026-07-17 15:10:16.068716163 +0000 UTC m=+66.343864482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-htmm8" (UniqueName: "kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8") pod "network-check-target-8v2c8" (UID: "03ec8278-12ed-4a29-b1cc-d3722d3a3b13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 15:09:45.234956 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.234923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:45.234956 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.234955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:45.235590 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.235077    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:45.235590 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:45.235082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smxd9" podUID="da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39"
Jul 17 15:09:45.235590 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:45.235155    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rrvds" podUID="e455437e-2d8c-452a-874d-287c57ebc381"
Jul 17 15:09:45.235590 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:45.235204    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v2c8" podUID="03ec8278-12ed-4a29-b1cc-d3722d3a3b13"
Jul 17 15:09:45.446104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.446040    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="a6944ee9ee737aa117e4078951760c0e3b5761d89a28cbb5b1dc8f312e014b39" exitCode=0
Jul 17 15:09:45.446104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.446110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"a6944ee9ee737aa117e4078951760c0e3b5761d89a28cbb5b1dc8f312e014b39"}
Jul 17 15:09:45.528345 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.528323    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-207.ec2.internal" event="NodeReady"
Jul 17 15:09:45.528467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.528440    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 15:09:45.577047 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.577022    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r8892"]
Jul 17 15:09:45.588702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.588667    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-44t9v"]
Jul 17 15:09:45.588840 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.588763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.590921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.590893    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m7zkr\""
Jul 17 15:09:45.591099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.590893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 15:09:45.591099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.590899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 15:09:45.600893 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.600869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8892"]
Jul 17 15:09:45.600987 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.600912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-44t9v"]
Jul 17 15:09:45.600987 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.600976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.603114 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.603095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 15:09:45.603218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.603125    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 15:09:45.603218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.603162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hhvk5\""
Jul 17 15:09:45.603218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.603097    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 15:09:45.679340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-kube-api-access-jv7qq\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.679509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.679509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03b96b1e-5919-432e-a238-47678e66b248-tmp-dir\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.679509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-cert\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.679509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv8s9\" (UniqueName: \"kubernetes.io/projected/03b96b1e-5919-432e-a238-47678e66b248-kube-api-access-fv8s9\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.679640 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.679515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96b1e-5919-432e-a238-47678e66b248-config-volume\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.780704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-kube-api-access-jv7qq\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.780704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.780704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03b96b1e-5919-432e-a238-47678e66b248-tmp-dir\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.780704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-cert\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.780998 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv8s9\" (UniqueName: \"kubernetes.io/projected/03b96b1e-5919-432e-a238-47678e66b248-kube-api-access-fv8s9\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.780998 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.780793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96b1e-5919-432e-a238-47678e66b248-config-volume\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.780998 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:45.780805    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 15:09:45.780998 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:45.780865    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls podName:03b96b1e-5919-432e-a238-47678e66b248 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:46.280847167 +0000 UTC m=+36.555995486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls") pod "dns-default-r8892" (UID: "03b96b1e-5919-432e-a238-47678e66b248") : secret "dns-default-metrics-tls" not found
Jul 17 15:09:45.781217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.781134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03b96b1e-5919-432e-a238-47678e66b248-tmp-dir\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.781372 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.781352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03b96b1e-5919-432e-a238-47678e66b248-config-volume\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.785548 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.785415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-cert\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.791846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.791821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv8s9\" (UniqueName: \"kubernetes.io/projected/03b96b1e-5919-432e-a238-47678e66b248-kube-api-access-fv8s9\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:45.792314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.792296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7qq\" (UniqueName: \"kubernetes.io/projected/6f2ea865-60f9-49d9-91ac-4acd4b2c685b-kube-api-access-jv7qq\") pod \"ingress-canary-44t9v\" (UID: \"6f2ea865-60f9-49d9-91ac-4acd4b2c685b\") " pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:45.960150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:45.960113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-44t9v"
Jul 17 15:09:46.145195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:46.145007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-44t9v"]
Jul 17 15:09:46.150898 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:09:46.150872    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2ea865_60f9_49d9_91ac_4acd4b2c685b.slice/crio-ccc275d0fe803ed9f525aebde745fca6fb1ac564b8888509732be3087f81f93c WatchSource:0}: Error finding container ccc275d0fe803ed9f525aebde745fca6fb1ac564b8888509732be3087f81f93c: Status 404 returned error can't find the container with id ccc275d0fe803ed9f525aebde745fca6fb1ac564b8888509732be3087f81f93c
Jul 17 15:09:46.286171 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:46.286135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:46.286576 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:46.286254    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 15:09:46.286576 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:46.286312    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls podName:03b96b1e-5919-432e-a238-47678e66b248 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:47.286296105 +0000 UTC m=+37.561444404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls") pod "dns-default-r8892" (UID: "03b96b1e-5919-432e-a238-47678e66b248") : secret "dns-default-metrics-tls" not found
Jul 17 15:09:46.449357 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:46.449261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-44t9v" event={"ID":"6f2ea865-60f9-49d9-91ac-4acd4b2c685b","Type":"ContainerStarted","Data":"ccc275d0fe803ed9f525aebde745fca6fb1ac564b8888509732be3087f81f93c"}
Jul 17 15:09:46.452530 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:46.452496    2566 generic.go:356] "Generic (PLEG): container finished" podID="303737d8-f0e3-4189-9a8f-31a72d3cd29a" containerID="f3c828f27fb6d264495b3dd959e85f6a061c1e56167589897d82e0f9430281bd" exitCode=0
Jul 17 15:09:46.452530 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:46.452528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerDied","Data":"f3c828f27fb6d264495b3dd959e85f6a061c1e56167589897d82e0f9430281bd"}
Jul 17 15:09:47.235282 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.235078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:09:47.235557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.235078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:09:47.235557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.235078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:09:47.238009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.237987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 15:09:47.238474 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.238452    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 15:09:47.238941 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.238920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54ptc\""
Jul 17 15:09:47.239027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.238926    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 15:09:47.239027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.238998    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 15:09:47.248120 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.248095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gqh\""
Jul 17 15:09:47.295714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.295677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:47.296142 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:47.295824    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 15:09:47.296142 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:47.295872    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls podName:03b96b1e-5919-432e-a238-47678e66b248 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:49.295858415 +0000 UTC m=+39.571006714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls") pod "dns-default-r8892" (UID: "03b96b1e-5919-432e-a238-47678e66b248") : secret "dns-default-metrics-tls" not found
Jul 17 15:09:47.458517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.458475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4g29s" event={"ID":"303737d8-f0e3-4189-9a8f-31a72d3cd29a","Type":"ContainerStarted","Data":"c2689318765f5f670725efa741fd120145857cee5cdcd95258f6f991efc97b0a"}
Jul 17 15:09:47.482964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:47.482914    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4g29s" podStartSLOduration=8.15668948 podStartE2EDuration="37.482898488s" totalImagesPullingTime="29.326209008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:47.482478901 +0000 UTC m=+37.757627213" watchObservedRunningTime="2026-07-17 15:09:47.482898488 +0000 UTC m=+37.758046809"
Jul 17 15:09:49.113640 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.113610    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zwst8"]
Jul 17 15:09:49.121516 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.121480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.123877 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.123857    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 15:09:49.124007 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.123880    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 15:09:49.124314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.124295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rtllk\""
Jul 17 15:09:49.124314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.124308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 15:09:49.124404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.124344    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 15:09:49.129911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.128352    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zwst8"]
Jul 17 15:09:49.211390 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.211339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jhz\" (UniqueName: \"kubernetes.io/projected/6eea7d47-c619-40e3-be9f-553b8df97479-kube-api-access-d2jhz\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.211390 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.211390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eea7d47-c619-40e3-be9f-553b8df97479-data-volume\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.211621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.211417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.211621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.211538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.211621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.211582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eea7d47-c619-40e3-be9f-553b8df97479-crio-socket\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jhz\" (UniqueName: \"kubernetes.io/projected/6eea7d47-c619-40e3-be9f-553b8df97479-kube-api-access-d2jhz\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eea7d47-c619-40e3-be9f-553b8df97479-data-volume\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312466    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312551    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:49.812530588 +0000 UTC m=+40.087678887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : configmap "kube-rbac-proxy" not found
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eea7d47-c619-40e3-be9f-553b8df97479-crio-socket\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312640    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:49.312714 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312696    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:49.812680531 +0000 UTC m=+40.087828831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:49.313100 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312759    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 15:09:49.313100 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.312799    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls podName:03b96b1e-5919-432e-a238-47678e66b248 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:53.312786456 +0000 UTC m=+43.587934758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls") pod "dns-default-r8892" (UID: "03b96b1e-5919-432e-a238-47678e66b248") : secret "dns-default-metrics-tls" not found
Jul 17 15:09:49.313100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eea7d47-c619-40e3-be9f-553b8df97479-crio-socket\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.313100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.312992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eea7d47-c619-40e3-be9f-553b8df97479-data-volume\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.327797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.327724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jhz\" (UniqueName: \"kubernetes.io/projected/6eea7d47-c619-40e3-be9f-553b8df97479-kube-api-access-d2jhz\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.465118 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.465083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-44t9v" event={"ID":"6f2ea865-60f9-49d9-91ac-4acd4b2c685b","Type":"ContainerStarted","Data":"5561d38f8701fe48691e468145b9813e7705bf1177e20f76aab36452a4d6fe3f"}
Jul 17 15:09:49.481615 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.481534    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-44t9v" podStartSLOduration=2.154016447 podStartE2EDuration="4.48151551s" totalImagesPullingTime="2.327499063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:09:49.481460183 +0000 UTC m=+39.756608505" watchObservedRunningTime="2026-07-17 15:09:49.48151551 +0000 UTC m=+39.756663832"
Jul 17 15:09:49.816921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.816875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.817127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:49.816951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:49.817127 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.817031    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 15:09:49.817127 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.817045    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:49.817127 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.817115    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:50.817100899 +0000 UTC m=+41.092249198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:49.817127 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:49.817129    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:50.81712303 +0000 UTC m=+41.092271328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : configmap "kube-rbac-proxy" not found
Jul 17 15:09:50.824421 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:50.824385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:50.825014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:50.824456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:50.825014 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:50.824531    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 15:09:50.825014 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:50.824553    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:50.825014 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:50.824596    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:52.824579232 +0000 UTC m=+43.099727531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : configmap "kube-rbac-proxy" not found
Jul 17 15:09:50.825014 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:50.824614    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:52.824607337 +0000 UTC m=+43.099755635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:50.914728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:50.914701    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hlbt_02bbe719-e77d-43ec-b9e4-9737732089aa/dns-node-resolver/0.log"
Jul 17 15:09:51.619558 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.619523    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:09:51.649076 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.649021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:09:51.649240 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.649138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.651352 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.651321    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 15:09:51.651594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.651577    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hgpkw\""
Jul 17 15:09:51.651814 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.651797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 15:09:51.651878 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.651864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 15:09:51.659050 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.659024    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 15:09:51.831859 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.831818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.831867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.831897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.831968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.832039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.832087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpk5\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.832130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.832241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.832149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.909928 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.909862    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tx5z_e9e5939e-d8b3-4860-857d-e7d6b6726d31/node-ca/0.log"
Jul 17 15:09:51.933231 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933620 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpk5\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933620 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.933527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.933738 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:51.933634    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 15:09:51.933738 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:51.933649    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687fff5cd9-7h6tm: secret "image-registry-tls" not found
Jul 17 15:09:51.933738 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:51.933710    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls podName:db12401d-a044-4952-a0d8-077a8d4ac28d nodeName:}" failed. No retries permitted until 2026-07-17 15:09:52.43368905 +0000 UTC m=+42.708837351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls") pod "image-registry-687fff5cd9-7h6tm" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d") : secret "image-registry-tls" not found
Jul 17 15:09:51.934042 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.934011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.934265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.934204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.936037 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.936014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.936205 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.936186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.942341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.942317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.943091 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.943071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpk5\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:51.945961 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:51.945945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:52.436747 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:52.436711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:52.436923 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.436839    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 15:09:52.436923 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.436853    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687fff5cd9-7h6tm: secret "image-registry-tls" not found
Jul 17 15:09:52.436923 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.436904    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls podName:db12401d-a044-4952-a0d8-077a8d4ac28d nodeName:}" failed. No retries permitted until 2026-07-17 15:09:53.436889923 +0000 UTC m=+43.712038223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls") pod "image-registry-687fff5cd9-7h6tm" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d") : secret "image-registry-tls" not found
Jul 17 15:09:52.512909 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:52.512870    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-44t9v_6f2ea865-60f9-49d9-91ac-4acd4b2c685b/serve-healthcheck-canary/0.log"
Jul 17 15:09:52.839141 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:52.839107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:52.839533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:52.839178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:52.839533 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.839239    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 15:09:52.839533 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.839284    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:52.839533 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.839303    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:56.839288383 +0000 UTC m=+47.114436682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : configmap "kube-rbac-proxy" not found
Jul 17 15:09:52.839533 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:52.839330    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:09:56.83931919 +0000 UTC m=+47.114467488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:53.343308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:53.343271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:09:53.343487 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:53.343433    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 15:09:53.343528 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:53.343496    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls podName:03b96b1e-5919-432e-a238-47678e66b248 nodeName:}" failed. No retries permitted until 2026-07-17 15:10:01.343478977 +0000 UTC m=+51.618627278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls") pod "dns-default-r8892" (UID: "03b96b1e-5919-432e-a238-47678e66b248") : secret "dns-default-metrics-tls" not found
Jul 17 15:09:53.444492 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:53.444459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:53.444627 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:53.444612    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 15:09:53.444672 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:53.444630    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687fff5cd9-7h6tm: secret "image-registry-tls" not found
Jul 17 15:09:53.444701 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:53.444683    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls podName:db12401d-a044-4952-a0d8-077a8d4ac28d nodeName:}" failed. No retries permitted until 2026-07-17 15:09:55.444665911 +0000 UTC m=+45.719814210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls") pod "image-registry-687fff5cd9-7h6tm" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d") : secret "image-registry-tls" not found
Jul 17 15:09:55.460885 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:55.460849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:55.461303 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:55.460974    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 15:09:55.461303 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:55.460986    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687fff5cd9-7h6tm: secret "image-registry-tls" not found
Jul 17 15:09:55.461303 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:55.461035    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls podName:db12401d-a044-4952-a0d8-077a8d4ac28d nodeName:}" failed. No retries permitted until 2026-07-17 15:09:59.461022517 +0000 UTC m=+49.736170815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls") pod "image-registry-687fff5cd9-7h6tm" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d") : secret "image-registry-tls" not found
Jul 17 15:09:56.872612 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:56.872569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:56.873119 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:56.872637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:56.873119 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:56.872772    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:56.873119 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:56.872873    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls podName:6eea7d47-c619-40e3-be9f-553b8df97479 nodeName:}" failed. No retries permitted until 2026-07-17 15:10:04.872850121 +0000 UTC m=+55.147998436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zwst8" (UID: "6eea7d47-c619-40e3-be9f-553b8df97479") : secret "insights-runtime-extractor-tls" not found
Jul 17 15:09:56.873315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:56.873132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eea7d47-c619-40e3-be9f-553b8df97479-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:09:59.493870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:09:59.493840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:09:59.494299 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:59.493953    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 15:09:59.494299 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:59.493966    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687fff5cd9-7h6tm: secret "image-registry-tls" not found
Jul 17 15:09:59.494299 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:09:59.494014    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls podName:db12401d-a044-4952-a0d8-077a8d4ac28d nodeName:}" failed. No retries permitted until 2026-07-17 15:10:07.493999541 +0000 UTC m=+57.769147840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls") pod "image-registry-687fff5cd9-7h6tm" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d") : secret "image-registry-tls" not found
Jul 17 15:10:01.407452 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:01.407408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:10:01.409912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:01.409886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03b96b1e-5919-432e-a238-47678e66b248-metrics-tls\") pod \"dns-default-r8892\" (UID: \"03b96b1e-5919-432e-a238-47678e66b248\") " pod="openshift-dns/dns-default-r8892"
Jul 17 15:10:01.500613 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:01.500586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r8892"
Jul 17 15:10:01.671135 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:01.671016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r8892"]
Jul 17 15:10:01.676567 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:01.676520    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b96b1e_5919_432e_a238_47678e66b248.slice/crio-ed7b24c20f0892a9b3883cb740eecce2f161cb6f3fe66a196138343ddf575bf0 WatchSource:0}: Error finding container ed7b24c20f0892a9b3883cb740eecce2f161cb6f3fe66a196138343ddf575bf0: Status 404 returned error can't find the container with id ed7b24c20f0892a9b3883cb740eecce2f161cb6f3fe66a196138343ddf575bf0
Jul 17 15:10:02.498690 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:02.498644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8892" event={"ID":"03b96b1e-5919-432e-a238-47678e66b248","Type":"ContainerStarted","Data":"ed7b24c20f0892a9b3883cb740eecce2f161cb6f3fe66a196138343ddf575bf0"}
Jul 17 15:10:04.932673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:04.932639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:10:04.935237 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:04.935217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eea7d47-c619-40e3-be9f-553b8df97479-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zwst8\" (UID: \"6eea7d47-c619-40e3-be9f-553b8df97479\") " pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:10:05.034075 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.034017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zwst8"
Jul 17 15:10:05.163529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.163488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zwst8"]
Jul 17 15:10:05.168649 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:05.168618    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eea7d47_c619_40e3_be9f_553b8df97479.slice/crio-f74acfb339bad7bb3787584175921a0a8ac6f7647e74985e30333f34dec443cc WatchSource:0}: Error finding container f74acfb339bad7bb3787584175921a0a8ac6f7647e74985e30333f34dec443cc: Status 404 returned error can't find the container with id f74acfb339bad7bb3787584175921a0a8ac6f7647e74985e30333f34dec443cc
Jul 17 15:10:05.507681 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.507646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8892" event={"ID":"03b96b1e-5919-432e-a238-47678e66b248","Type":"ContainerStarted","Data":"cb85dcf7f83e80c45d94734057f63bb2bf857d33b2d1d9481971a649e3ca5fbf"}
Jul 17 15:10:05.507845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.507707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r8892" event={"ID":"03b96b1e-5919-432e-a238-47678e66b248","Type":"ContainerStarted","Data":"3d43bdf65638389fcb66332791c785488a6b512995f65871d926c61e5ffc590c"}
Jul 17 15:10:05.507845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.507744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r8892"
Jul 17 15:10:05.508764 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.508735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwst8" event={"ID":"6eea7d47-c619-40e3-be9f-553b8df97479","Type":"ContainerStarted","Data":"f74acfb339bad7bb3787584175921a0a8ac6f7647e74985e30333f34dec443cc"}
Jul 17 15:10:05.528704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:05.528654    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r8892" podStartSLOduration=17.759615434 podStartE2EDuration="20.528638666s" totalImagesPullingTime="2.769023232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:05.52801911 +0000 UTC m=+55.803167430" watchObservedRunningTime="2026-07-17 15:10:05.528638666 +0000 UTC m=+55.803786977"
Jul 17 15:10:07.560045 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:07.560004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:07.562945 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:07.562912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"image-registry-687fff5cd9-7h6tm\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") " pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:07.862374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:07.862264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:10.387633 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.387602    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:10:10.398166 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:10.398137    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb12401d_a044_4952_a0d8_077a8d4ac28d.slice/crio-0492a6a6cbf654848722350f4c5db5e1105c42cd360927c0b47c16b1e3fca772 WatchSource:0}: Error finding container 0492a6a6cbf654848722350f4c5db5e1105c42cd360927c0b47c16b1e3fca772: Status 404 returned error can't find the container with id 0492a6a6cbf654848722350f4c5db5e1105c42cd360927c0b47c16b1e3fca772
Jul 17 15:10:10.453709 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.453683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxh8w"
Jul 17 15:10:10.523539 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.523465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" event={"ID":"db12401d-a044-4952-a0d8-077a8d4ac28d","Type":"ContainerStarted","Data":"0492a6a6cbf654848722350f4c5db5e1105c42cd360927c0b47c16b1e3fca772"}
Jul 17 15:10:10.673683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.673652    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"]
Jul 17 15:10:10.684651 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.684629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.687171 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.687147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"]
Jul 17 15:10:10.687437 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.687414    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 15:10:10.687976 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.687544    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 15:10:10.687976 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.687549    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 15:10:10.687976 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.687831    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 15:10:10.692540 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.692519    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:10:10.733119 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.733090    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54d59b885f-j4dc5"]
Jul 17 15:10:10.757543 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.757505    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54d59b885f-j4dc5"]
Jul 17 15:10:10.757706 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.757627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.774351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.774284    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-rnd2q"]
Jul 17 15:10:10.782580 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.782553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:10.782862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.782839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-certificates\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.782937 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.782879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f24e56c6-84c5-475d-acff-75f6040ca00b-ca-trust-extracted\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.782989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.782937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcsh\" (UniqueName: \"kubernetes.io/projected/403e3a68-e973-455b-b7b8-69575a89133c-kube-api-access-mfcsh\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.782989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.782956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-bound-sa-token\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.783101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xd6k\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-kube-api-access-7xd6k\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.783101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/403e3a68-e973-455b-b7b8-69575a89133c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.783101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-tls\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.783385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-trusted-ca\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.783385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403e3a68-e973-455b-b7b8-69575a89133c-tmp\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.783385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-image-registry-private-configuration\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.783385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.783200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-installation-pull-secrets\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.786121 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.786094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p9r85\""
Jul 17 15:10:10.786262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.786166    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 15:10:10.786262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.786246    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 15:10:10.790196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.790170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rnd2q"]
Jul 17 15:10:10.884288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcsh\" (UniqueName: \"kubernetes.io/projected/403e3a68-e973-455b-b7b8-69575a89133c-kube-api-access-mfcsh\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-bound-sa-token\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xd6k\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-kube-api-access-7xd6k\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/403e3a68-e973-455b-b7b8-69575a89133c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqm76\" (UniqueName: \"kubernetes.io/projected/b3e09dc3-9031-43cb-bd47-ca5066bc0579-kube-api-access-vqm76\") pod \"downloads-5bc67dcd46-rnd2q\" (UID: \"b3e09dc3-9031-43cb-bd47-ca5066bc0579\") " pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-tls\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.884497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-trusted-ca\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.884737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403e3a68-e973-455b-b7b8-69575a89133c-tmp\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.884860 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-image-registry-private-configuration\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.884916 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.884900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-installation-pull-secrets\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.885021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.885002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-certificates\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.885206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.885186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f24e56c6-84c5-475d-acff-75f6040ca00b-ca-trust-extracted\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.888174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.885371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/403e3a68-e973-455b-b7b8-69575a89133c-tmp\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.888174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.885649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f24e56c6-84c5-475d-acff-75f6040ca00b-ca-trust-extracted\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.888174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.886618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-certificates\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.888174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.887857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f24e56c6-84c5-475d-acff-75f6040ca00b-trusted-ca\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.891554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.888555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-registry-tls\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.891554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.888559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-installation-pull-secrets\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.891554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.889832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/403e3a68-e973-455b-b7b8-69575a89133c-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.891554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.890534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f24e56c6-84c5-475d-acff-75f6040ca00b-image-registry-private-configuration\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.896806 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.896775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcsh\" (UniqueName: \"kubernetes.io/projected/403e3a68-e973-455b-b7b8-69575a89133c-kube-api-access-mfcsh\") pod \"klusterlet-addon-workmgr-7f46dcb569-vhjhm\" (UID: \"403e3a68-e973-455b-b7b8-69575a89133c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:10.897037 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.897013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xd6k\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-kube-api-access-7xd6k\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.897141 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.897126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f24e56c6-84c5-475d-acff-75f6040ca00b-bound-sa-token\") pod \"image-registry-54d59b885f-j4dc5\" (UID: \"f24e56c6-84c5-475d-acff-75f6040ca00b\") " pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:10.985749 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.985712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqm76\" (UniqueName: \"kubernetes.io/projected/b3e09dc3-9031-43cb-bd47-ca5066bc0579-kube-api-access-vqm76\") pod \"downloads-5bc67dcd46-rnd2q\" (UID: \"b3e09dc3-9031-43cb-bd47-ca5066bc0579\") " pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:10.995916 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.995883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqm76\" (UniqueName: \"kubernetes.io/projected/b3e09dc3-9031-43cb-bd47-ca5066bc0579-kube-api-access-vqm76\") pod \"downloads-5bc67dcd46-rnd2q\" (UID: \"b3e09dc3-9031-43cb-bd47-ca5066bc0579\") " pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:10.996111 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:10.996041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:11.067016 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.066983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:11.092335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.092304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:11.144465 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.144348    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"]
Jul 17 15:10:11.150611 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:11.150569    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403e3a68_e973_455b_b7b8_69575a89133c.slice/crio-f09807509213a7c4fdf0dba72baaba468550d52f26aaebb4e7f35ba885b4b4e9 WatchSource:0}: Error finding container f09807509213a7c4fdf0dba72baaba468550d52f26aaebb4e7f35ba885b4b4e9: Status 404 returned error can't find the container with id f09807509213a7c4fdf0dba72baaba468550d52f26aaebb4e7f35ba885b4b4e9
Jul 17 15:10:11.266675 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.266598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54d59b885f-j4dc5"]
Jul 17 15:10:11.272291 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:11.272255    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24e56c6_84c5_475d_acff_75f6040ca00b.slice/crio-ecfb5273ffc646ebbc84eed748092ef9e1bcd366593ea6674dd025d0d64bcf4c WatchSource:0}: Error finding container ecfb5273ffc646ebbc84eed748092ef9e1bcd366593ea6674dd025d0d64bcf4c: Status 404 returned error can't find the container with id ecfb5273ffc646ebbc84eed748092ef9e1bcd366593ea6674dd025d0d64bcf4c
Jul 17 15:10:11.287499 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.287264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rnd2q"]
Jul 17 15:10:11.290249 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:11.290182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e09dc3_9031_43cb_bd47_ca5066bc0579.slice/crio-4751b1568bee2d8c5e3a103787392c08d4bb084b383aaeb11fd8949f7986fe1c WatchSource:0}: Error finding container 4751b1568bee2d8c5e3a103787392c08d4bb084b383aaeb11fd8949f7986fe1c: Status 404 returned error can't find the container with id 4751b1568bee2d8c5e3a103787392c08d4bb084b383aaeb11fd8949f7986fe1c
Jul 17 15:10:11.529904 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.529866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwst8" event={"ID":"6eea7d47-c619-40e3-be9f-553b8df97479","Type":"ContainerStarted","Data":"c3002851b8d92b65188f28a1ace847e458e106ac843e71d006133db10b6a401a"}
Jul 17 15:10:11.532186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.532151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" event={"ID":"db12401d-a044-4952-a0d8-077a8d4ac28d","Type":"ContainerStarted","Data":"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"}
Jul 17 15:10:11.532485 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.532452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:11.534300 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.534272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rnd2q" event={"ID":"b3e09dc3-9031-43cb-bd47-ca5066bc0579","Type":"ContainerStarted","Data":"4751b1568bee2d8c5e3a103787392c08d4bb084b383aaeb11fd8949f7986fe1c"}
Jul 17 15:10:11.535901 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.535876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm" event={"ID":"403e3a68-e973-455b-b7b8-69575a89133c","Type":"ContainerStarted","Data":"f09807509213a7c4fdf0dba72baaba468550d52f26aaebb4e7f35ba885b4b4e9"}
Jul 17 15:10:11.540240 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.540212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5" event={"ID":"f24e56c6-84c5-475d-acff-75f6040ca00b","Type":"ContainerStarted","Data":"a6f0fae892524052ef41febe599851650ece7fd77b6c9fad5039100e78213217"}
Jul 17 15:10:11.540359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.540257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5" event={"ID":"f24e56c6-84c5-475d-acff-75f6040ca00b","Type":"ContainerStarted","Data":"ecfb5273ffc646ebbc84eed748092ef9e1bcd366593ea6674dd025d0d64bcf4c"}
Jul 17 15:10:11.540419 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.540369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:11.558635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.558528    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" podStartSLOduration=20.558510816 podStartE2EDuration="20.558510816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:11.55726609 +0000 UTC m=+61.832414412" watchObservedRunningTime="2026-07-17 15:10:11.558510816 +0000 UTC m=+61.833659136"
Jul 17 15:10:11.577557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:11.577493    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5" podStartSLOduration=1.577467664 podStartE2EDuration="1.577467664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:11.576242515 +0000 UTC m=+61.851390838" watchObservedRunningTime="2026-07-17 15:10:11.577467664 +0000 UTC m=+61.852615985"
Jul 17 15:10:15.514355 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:15.514326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r8892"
Jul 17 15:10:15.935279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:15.935193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:10:15.937576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:15.937540    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 15:10:15.948925 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:15.948889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e455437e-2d8c-452a-874d-287c57ebc381-metrics-certs\") pod \"network-metrics-daemon-rrvds\" (UID: \"e455437e-2d8c-452a-874d-287c57ebc381\") " pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:10:16.035767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.035721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:10:16.037897 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.037873    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 15:10:16.049094 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.049043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gqh\""
Jul 17 15:10:16.049388 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.049357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39-original-pull-secret\") pod \"global-pull-secret-syncer-smxd9\" (UID: \"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39\") " pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:10:16.056001 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.055778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smxd9"
Jul 17 15:10:16.058320 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.058265    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rrvds"
Jul 17 15:10:16.136230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.136186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:10:16.138534 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.138510    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 15:10:16.149127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.148905    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 15:10:16.160594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.160569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmm8\" (UniqueName: \"kubernetes.io/projected/03ec8278-12ed-4a29-b1cc-d3722d3a3b13-kube-api-access-htmm8\") pod \"network-check-target-8v2c8\" (UID: \"03ec8278-12ed-4a29-b1cc-d3722d3a3b13\") " pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:10:16.363964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.363923    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-54ptc\""
Jul 17 15:10:16.372742 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:16.372508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:10:17.880110 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:17.880013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rrvds"]
Jul 17 15:10:17.905737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:17.905704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smxd9"]
Jul 17 15:10:18.021677 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:18.021641    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode455437e_2d8c_452a_874d_287c57ebc381.slice/crio-3d0390337ff94c63bda598875ba0aef05692568799f21869223758c50a534d36 WatchSource:0}: Error finding container 3d0390337ff94c63bda598875ba0aef05692568799f21869223758c50a534d36: Status 404 returned error can't find the container with id 3d0390337ff94c63bda598875ba0aef05692568799f21869223758c50a534d36
Jul 17 15:10:18.022905 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:18.022832    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5436cb_fb18_4ba3_a0d2_f4db7dcc7f39.slice/crio-e7a991e87835353bc8df96b720ff30eaa7268011fd5519c8b250d14ccc432333 WatchSource:0}: Error finding container e7a991e87835353bc8df96b720ff30eaa7268011fd5519c8b250d14ccc432333: Status 404 returned error can't find the container with id e7a991e87835353bc8df96b720ff30eaa7268011fd5519c8b250d14ccc432333
Jul 17 15:10:18.110042 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.110009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8v2c8"]
Jul 17 15:10:18.113872 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:18.113842    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ec8278_12ed_4a29_b1cc_d3722d3a3b13.slice/crio-14e07d2c0c3c0b139d6f543e0124f58c5c712bef508cd9df1a216b96934c5d4a WatchSource:0}: Error finding container 14e07d2c0c3c0b139d6f543e0124f58c5c712bef508cd9df1a216b96934c5d4a: Status 404 returned error can't find the container with id 14e07d2c0c3c0b139d6f543e0124f58c5c712bef508cd9df1a216b96934c5d4a
Jul 17 15:10:18.573789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.573741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8v2c8" event={"ID":"03ec8278-12ed-4a29-b1cc-d3722d3a3b13","Type":"ContainerStarted","Data":"14e07d2c0c3c0b139d6f543e0124f58c5c712bef508cd9df1a216b96934c5d4a"}
Jul 17 15:10:18.577791 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.577748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwst8" event={"ID":"6eea7d47-c619-40e3-be9f-553b8df97479","Type":"ContainerStarted","Data":"4c0b61c7d5add7cce0ed553c41833f5aa81c31b9f5acea13ae5c3449c3b1bb21"}
Jul 17 15:10:18.581311 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.581269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm" event={"ID":"403e3a68-e973-455b-b7b8-69575a89133c","Type":"ContainerStarted","Data":"8a43bd281911dcaf3dfdcf3df8c5009e34e54a0d4a390ef7294cefa79b3c5f5c"}
Jul 17 15:10:18.582386 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.582363    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:18.584012 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.583968    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm"
Jul 17 15:10:18.586230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.586201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rrvds" event={"ID":"e455437e-2d8c-452a-874d-287c57ebc381","Type":"ContainerStarted","Data":"3d0390337ff94c63bda598875ba0aef05692568799f21869223758c50a534d36"}
Jul 17 15:10:18.587579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.587516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smxd9" event={"ID":"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39","Type":"ContainerStarted","Data":"e7a991e87835353bc8df96b720ff30eaa7268011fd5519c8b250d14ccc432333"}
Jul 17 15:10:18.606307 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:18.606247    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f46dcb569-vhjhm" podStartSLOduration=2.036996405 podStartE2EDuration="8.606228152s" totalImagesPullingTime="6.569231747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:18.605869633 +0000 UTC m=+68.881017954" watchObservedRunningTime="2026-07-17 15:10:18.606228152 +0000 UTC m=+68.881376473"
Jul 17 15:10:19.597779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:19.597663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rrvds" event={"ID":"e455437e-2d8c-452a-874d-287c57ebc381","Type":"ContainerStarted","Data":"546039d9b99cfb573a27479bbd7a85a6d42b9563f804d8bd5b981fd3afe1cf81"}
Jul 17 15:10:19.597779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:19.597719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rrvds" event={"ID":"e455437e-2d8c-452a-874d-287c57ebc381","Type":"ContainerStarted","Data":"76ce896029fdc19f8ea4d4190c672661c8a4018c342cea8a54a47e2cf73c0f81"}
Jul 17 15:10:19.616281 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:19.615923    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rrvds" podStartSLOduration=68.368830683 podStartE2EDuration="1m9.615887181s" totalImagesPullingTime="1.247056498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:19.613738605 +0000 UTC m=+69.888886927" watchObservedRunningTime="2026-07-17 15:10:19.615887181 +0000 UTC m=+69.891035502"
Jul 17 15:10:20.690092 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.690037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:10:20.693653 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.692440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.701957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.701319    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5mff\""
Jul 17 15:10:20.701957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.701567    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 15:10:20.702143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.701957    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 15:10:20.702448 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.702254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 15:10:20.712189 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.711269    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 15:10:20.712189 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.711297    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 15:10:20.718124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.718093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:10:20.723941 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.723909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.774921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkls2\" (UniqueName: \"kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.775350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.775222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876091 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876091 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkls2\" (UniqueName: \"kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.876339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.876225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.877502 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.877470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.877659 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.877602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.878051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.877991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.878051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.878000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.881515 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.881472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.884877 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.884822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:20.890009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:20.889957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkls2\" (UniqueName: \"kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2\") pod \"console-5d9798548f-jjk45\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") " pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:21.022138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:21.022105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:24.205405 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.205365    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"]
Jul 17 15:10:24.207834 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.207811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.210797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.210767    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 15:10:24.210935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.210809    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 15:10:24.210935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.210888    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 15:10:24.210935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.210815    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 15:10:24.211090 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.210767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pcbwv\""
Jul 17 15:10:24.211138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.211111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 15:10:24.219000 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.218978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"]
Jul 17 15:10:24.302690 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.302652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfpq\" (UniqueName: \"kubernetes.io/projected/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-kube-api-access-9zfpq\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.302868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.302701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.302868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.302776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.302868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.302823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.403957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.403910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.403957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.403964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.404213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.403998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.404213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.404043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfpq\" (UniqueName: \"kubernetes.io/projected/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-kube-api-access-9zfpq\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.404213 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:24.404157    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 17 15:10:24.404348 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:24.404237    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls podName:33b6c420-8e69-4f03-87fd-7d6f37dfb25f nodeName:}" failed. No retries permitted until 2026-07-17 15:10:24.904214917 +0000 UTC m=+75.179363222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-jq9jm" (UID: "33b6c420-8e69-4f03-87fd-7d6f37dfb25f") : secret "prometheus-operator-tls" not found
Jul 17 15:10:24.404919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.404894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.406994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.406967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.415968 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.415944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfpq\" (UniqueName: \"kubernetes.io/projected/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-kube-api-access-9zfpq\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.908463 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.908419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:24.911183 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:24.911153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/33b6c420-8e69-4f03-87fd-7d6f37dfb25f-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-jq9jm\" (UID: \"33b6c420-8e69-4f03-87fd-7d6f37dfb25f\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:25.118707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:25.118668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"
Jul 17 15:10:31.539491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:31.539354    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:32.550795 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:32.550759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54d59b885f-j4dc5"
Jul 17 15:10:33.306107 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.306080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:10:33.311376 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:33.311351    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c9987d_79b7_47ab_857d_3f9d490f998a.slice/crio-ac3c59a0f01d90b006ab191c04e97b07c48415f3e5a8a59cff4797ff56cc0a4f WatchSource:0}: Error finding container ac3c59a0f01d90b006ab191c04e97b07c48415f3e5a8a59cff4797ff56cc0a4f: Status 404 returned error can't find the container with id ac3c59a0f01d90b006ab191c04e97b07c48415f3e5a8a59cff4797ff56cc0a4f
Jul 17 15:10:33.324865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.324835    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm"]
Jul 17 15:10:33.328832 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:33.328765    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b6c420_8e69_4f03_87fd_7d6f37dfb25f.slice/crio-4f88e78ec015366a4be1831f33e81ca027a64013d8f4fbfd4a884a3b23b33b7e WatchSource:0}: Error finding container 4f88e78ec015366a4be1831f33e81ca027a64013d8f4fbfd4a884a3b23b33b7e: Status 404 returned error can't find the container with id 4f88e78ec015366a4be1831f33e81ca027a64013d8f4fbfd4a884a3b23b33b7e
Jul 17 15:10:33.654677 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.654634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm" event={"ID":"33b6c420-8e69-4f03-87fd-7d6f37dfb25f","Type":"ContainerStarted","Data":"4f88e78ec015366a4be1831f33e81ca027a64013d8f4fbfd4a884a3b23b33b7e"}
Jul 17 15:10:33.657513 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.657475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zwst8" event={"ID":"6eea7d47-c619-40e3-be9f-553b8df97479","Type":"ContainerStarted","Data":"37bd782891ca66519d5e7a4e1f24dc3b433ad49f915ba73fe1ff6e4bc4936af2"}
Jul 17 15:10:33.660227 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.660202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rnd2q" event={"ID":"b3e09dc3-9031-43cb-bd47-ca5066bc0579","Type":"ContainerStarted","Data":"e611293bb614fb89188a547121a390ed73449dd106a70ca27010e089a25a8b18"}
Jul 17 15:10:33.660416 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.660392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:33.662520 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.662168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smxd9" event={"ID":"da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39","Type":"ContainerStarted","Data":"dff4e15d505eda93b7a36d1f9455ef58a80995d192f50f5c983dca247b2d06c0"}
Jul 17 15:10:33.664117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.664090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8v2c8" event={"ID":"03ec8278-12ed-4a29-b1cc-d3722d3a3b13","Type":"ContainerStarted","Data":"73f08038291a623b37cdffaaf055da06b207eff0dcc5cbc0dc3496f991c87422"}
Jul 17 15:10:33.664221 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.664156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:10:33.665589 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.665563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9798548f-jjk45" event={"ID":"73c9987d-79b7-47ab-857d-3f9d490f998a","Type":"ContainerStarted","Data":"ac3c59a0f01d90b006ab191c04e97b07c48415f3e5a8a59cff4797ff56cc0a4f"}
Jul 17 15:10:33.675790 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.675759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-rnd2q"
Jul 17 15:10:33.678175 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.678124    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zwst8" podStartSLOduration=16.992785181 podStartE2EDuration="44.678106453s" totalImagesPullingTime="27.685321272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:33.677300445 +0000 UTC m=+83.952448767" watchObservedRunningTime="2026-07-17 15:10:33.678106453 +0000 UTC m=+83.953254778"
Jul 17 15:10:33.696103 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.696008    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-smxd9" podStartSLOduration=68.611958302 podStartE2EDuration="1m23.695986409s" totalImagesPullingTime="15.084028107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:33.69500193 +0000 UTC m=+83.970150250" watchObservedRunningTime="2026-07-17 15:10:33.695986409 +0000 UTC m=+83.971134731"
Jul 17 15:10:33.713813 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:33.713752    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8v2c8" podStartSLOduration=68.708780164 podStartE2EDuration="1m23.713732529s" totalImagesPullingTime="15.004952365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:33.71331615 +0000 UTC m=+83.988464503" watchObservedRunningTime="2026-07-17 15:10:33.713732529 +0000 UTC m=+83.988880850"
Jul 17 15:10:36.563600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:36.563517    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" podUID="db12401d-a044-4952-a0d8-077a8d4ac28d" containerName="registry" containerID="cri-o://209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2" gracePeriod=30
Jul 17 15:10:37.210193 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.209886    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:37.232569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.232400    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-rnd2q" podStartSLOduration=5.336474831 podStartE2EDuration="27.232379052s" totalImagesPullingTime="21.895904221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:33.733374376 +0000 UTC m=+84.008522701" watchObservedRunningTime="2026-07-17 15:10:37.232379052 +0000 UTC m=+87.507527374"
Jul 17 15:10:37.319028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.318946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319183    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.319276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tpk5\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5\") pod \"db12401d-a044-4952-a0d8-077a8d4ac28d\" (UID: \"db12401d-a044-4952-a0d8-077a8d4ac28d\") "
Jul 17 15:10:37.322932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.319899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:10:37.322932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.320043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:10:37.324170 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.324113    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:10:37.324757 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.324723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:10:37.325316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.325279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:10:37.325316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.325283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:10:37.325739 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.325715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5" (OuterVolumeSpecName: "kube-api-access-7tpk5") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "kube-api-access-7tpk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:10:37.329458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.329421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "db12401d-a044-4952-a0d8-077a8d4ac28d" (UID: "db12401d-a044-4952-a0d8-077a8d4ac28d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:10:37.420708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420673    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpk5\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-kube-api-access-7tpk5\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420711    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420729    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-image-registry-private-configuration\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420744    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-trusted-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420757    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db12401d-a044-4952-a0d8-077a8d4ac28d-registry-certificates\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420773    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db12401d-a044-4952-a0d8-077a8d4ac28d-ca-trust-extracted\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420784    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db12401d-a044-4952-a0d8-077a8d4ac28d-installation-pull-secrets\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.420900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.420794    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db12401d-a044-4952-a0d8-077a8d4ac28d-bound-sa-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:10:37.694826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.694742    2566 generic.go:356] "Generic (PLEG): container finished" podID="db12401d-a044-4952-a0d8-077a8d4ac28d" containerID="209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2" exitCode=0
Jul 17 15:10:37.694826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.694818    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm"
Jul 17 15:10:37.695375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.694874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" event={"ID":"db12401d-a044-4952-a0d8-077a8d4ac28d","Type":"ContainerDied","Data":"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"}
Jul 17 15:10:37.695375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.694952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-687fff5cd9-7h6tm" event={"ID":"db12401d-a044-4952-a0d8-077a8d4ac28d","Type":"ContainerDied","Data":"0492a6a6cbf654848722350f4c5db5e1105c42cd360927c0b47c16b1e3fca772"}
Jul 17 15:10:37.695375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.694984    2566 scope.go:117] "RemoveContainer" containerID="209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"
Jul 17 15:10:37.698386 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.698336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9798548f-jjk45" event={"ID":"73c9987d-79b7-47ab-857d-3f9d490f998a","Type":"ContainerStarted","Data":"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"}
Jul 17 15:10:37.700661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.700632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm" event={"ID":"33b6c420-8e69-4f03-87fd-7d6f37dfb25f","Type":"ContainerStarted","Data":"ed0da02e637b0c6d29b8a7a6a340d95348390f4d3bc09fa73be5ffaaf7ccdf23"}
Jul 17 15:10:37.700772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.700682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm" event={"ID":"33b6c420-8e69-4f03-87fd-7d6f37dfb25f","Type":"ContainerStarted","Data":"a5b714b49f30d12cf39632d07b07b9100e32302c6876711bbae08d2872c7e6a3"}
Jul 17 15:10:37.706540 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.706520    2566 scope.go:117] "RemoveContainer" containerID="209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"
Jul 17 15:10:37.706881 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:37.706849    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2\": container with ID starting with 209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2 not found: ID does not exist" containerID="209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"
Jul 17 15:10:37.706967 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.706889    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2"} err="failed to get container status \"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2\": rpc error: code = NotFound desc = could not find container \"209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2\": container with ID starting with 209d27fe59617a703c571f1affbd059c113ba0b036b2bea4018b4bdbda9cb1d2 not found: ID does not exist"
Jul 17 15:10:37.724708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.724652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9798548f-jjk45" podStartSLOduration=13.661922103 podStartE2EDuration="17.72463219s" totalImagesPullingTime="4.062710087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:37.723266976 +0000 UTC m=+87.998415297" watchObservedRunningTime="2026-07-17 15:10:37.72463219 +0000 UTC m=+87.999780513"
Jul 17 15:10:37.771404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.771342    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-jq9jm" podStartSLOduration=10.098796265 podStartE2EDuration="13.771319617s" totalImagesPullingTime="3.672523352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:37.7477806 +0000 UTC m=+88.022928949" watchObservedRunningTime="2026-07-17 15:10:37.771319617 +0000 UTC m=+88.046467939"
Jul 17 15:10:37.773030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.772998    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:10:37.782796 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:37.782762    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-687fff5cd9-7h6tm"]
Jul 17 15:10:38.240856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:38.240824    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db12401d-a044-4952-a0d8-077a8d4ac28d" path="/var/lib/kubelet/pods/db12401d-a044-4952-a0d8-077a8d4ac28d/volumes"
Jul 17 15:10:39.559328 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.559291    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"]
Jul 17 15:10:39.559918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.559609    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db12401d-a044-4952-a0d8-077a8d4ac28d" containerName="registry"
Jul 17 15:10:39.559918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.559627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db12401d-a044-4952-a0d8-077a8d4ac28d" containerName="registry"
Jul 17 15:10:39.559918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.559684    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db12401d-a044-4952-a0d8-077a8d4ac28d" containerName="registry"
Jul 17 15:10:39.607901 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.607863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"]
Jul 17 15:10:39.608101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.607923    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75vzx"]
Jul 17 15:10:39.608169 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.608119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.610729 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.610702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 17 15:10:39.610910 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.610891    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 17 15:10:39.611041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.611004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ldgws\""
Jul 17 15:10:39.629847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.629808    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dg4kt"]
Jul 17 15:10:39.630039 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.629942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.634295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.634267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 17 15:10:39.634459 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.634398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 17 15:10:39.634565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.634541    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p6shd\""
Jul 17 15:10:39.634858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.634843    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 17 15:10:39.648845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.648807    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75vzx"]
Jul 17 15:10:39.649014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.648934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.652199 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.652174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 15:10:39.652373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.652214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wtgbr\""
Jul 17 15:10:39.652373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.652337    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 15:10:39.652515 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.652493    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 15:10:39.738731 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.738912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.738912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-tls\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.738912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-wtmp\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.738912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjmhv\" (UniqueName: \"kubernetes.io/projected/71506a20-0b90-4314-a750-f44ebb0a855b-kube-api-access-rjmhv\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.739143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc725\" (UniqueName: \"kubernetes.io/projected/2b11a47b-d852-4e1a-a881-7ea8050c6562-kube-api-access-cc725\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.739143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.738973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.739143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a47b-d852-4e1a-a881-7ea8050c6562-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.739143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-textfile\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.739390 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-metrics-client-ca\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.739460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.739509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.739509 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-sys\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.739606 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.739606 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-root\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.739709 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.739765 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71506a20-0b90-4314-a750-f44ebb0a855b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.739814 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.739814 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.739797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsm7\" (UniqueName: \"kubernetes.io/projected/09d5b15f-e7f4-490a-9713-452b2ddf077c-kube-api-access-qlsm7\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.840428 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.840345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.840428 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.840394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.840956 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:39.840930    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 17 15:10:39.841112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.840971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-tls\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841112 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:39.841006    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls podName:71506a20-0b90-4314-a750-f44ebb0a855b nodeName:}" failed. No retries permitted until 2026-07-17 15:10:40.340985966 +0000 UTC m=+90.616134278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-75vzx" (UID: "71506a20-0b90-4314-a750-f44ebb0a855b") : secret "kube-state-metrics-tls" not found
Jul 17 15:10:39.841112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-wtmp\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjmhv\" (UniqueName: \"kubernetes.io/projected/71506a20-0b90-4314-a750-f44ebb0a855b-kube-api-access-rjmhv\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.841351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc725\" (UniqueName: \"kubernetes.io/projected/2b11a47b-d852-4e1a-a881-7ea8050c6562-kube-api-access-cc725\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.841351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.841351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a47b-d852-4e1a-a881-7ea8050c6562-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.841351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-wtmp\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-textfile\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-metrics-client-ca\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-sys\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.841611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-root\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.842156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.842156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71506a20-0b90-4314-a750-f44ebb0a855b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.842156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.842156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.841707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsm7\" (UniqueName: \"kubernetes.io/projected/09d5b15f-e7f4-490a-9713-452b2ddf077c-kube-api-access-qlsm7\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.842156 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.842035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-sys\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.842442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.843302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.842526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-textfile\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.842950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-metrics-client-ca\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.843236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a47b-d852-4e1a-a881-7ea8050c6562-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.843999 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.843600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d5b15f-e7f4-490a-9713-452b2ddf077c-root\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843999 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.843751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843999 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.843926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-tls\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.843999 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.843949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71506a20-0b90-4314-a750-f44ebb0a855b-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.844386 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.844278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.844993 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.844614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d5b15f-e7f4-490a-9713-452b2ddf077c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.844993 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.844953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.846127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.846103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.846696 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.846671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b11a47b-d852-4e1a-a881-7ea8050c6562-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.855337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.855304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsm7\" (UniqueName: \"kubernetes.io/projected/09d5b15f-e7f4-490a-9713-452b2ddf077c-kube-api-access-qlsm7\") pod \"node-exporter-dg4kt\" (UID: \"09d5b15f-e7f4-490a-9713-452b2ddf077c\") " pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:39.855495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.855456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjmhv\" (UniqueName: \"kubernetes.io/projected/71506a20-0b90-4314-a750-f44ebb0a855b-kube-api-access-rjmhv\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:39.856164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.856138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc725\" (UniqueName: \"kubernetes.io/projected/2b11a47b-d852-4e1a-a881-7ea8050c6562-kube-api-access-cc725\") pod \"openshift-state-metrics-79f6958cdf-mgffm\" (UID: \"2b11a47b-d852-4e1a-a881-7ea8050c6562\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.920631 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.920590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"
Jul 17 15:10:39.960881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:39.960850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dg4kt"
Jul 17 15:10:40.006355 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:40.006306    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d5b15f_e7f4_490a_9713_452b2ddf077c.slice/crio-0eb49759cf45921d2cd428e84a67fa58f25bd0e5b33abe04be5b974fa81a508e WatchSource:0}: Error finding container 0eb49759cf45921d2cd428e84a67fa58f25bd0e5b33abe04be5b974fa81a508e: Status 404 returned error can't find the container with id 0eb49759cf45921d2cd428e84a67fa58f25bd0e5b33abe04be5b974fa81a508e
Jul 17 15:10:40.123427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.123338    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm"]
Jul 17 15:10:40.126796 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:40.126766    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b11a47b_d852_4e1a_a881_7ea8050c6562.slice/crio-1f8ee5360cca5c7f9c11a90cb4f35a5e96687a6ac55f03e997fd1504bcfa56f9 WatchSource:0}: Error finding container 1f8ee5360cca5c7f9c11a90cb4f35a5e96687a6ac55f03e997fd1504bcfa56f9: Status 404 returned error can't find the container with id 1f8ee5360cca5c7f9c11a90cb4f35a5e96687a6ac55f03e997fd1504bcfa56f9
Jul 17 15:10:40.345999 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:40.345561    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 17 15:10:40.345999 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:10:40.345641    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls podName:71506a20-0b90-4314-a750-f44ebb0a855b nodeName:}" failed. No retries permitted until 2026-07-17 15:10:41.345620716 +0000 UTC m=+91.620769022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-75vzx" (UID: "71506a20-0b90-4314-a750-f44ebb0a855b") : secret "kube-state-metrics-tls" not found
Jul 17 15:10:40.346357 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.346239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:40.716646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.716512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dg4kt" event={"ID":"09d5b15f-e7f4-490a-9713-452b2ddf077c","Type":"ContainerStarted","Data":"0eb49759cf45921d2cd428e84a67fa58f25bd0e5b33abe04be5b974fa81a508e"}
Jul 17 15:10:40.718870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.718829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm" event={"ID":"2b11a47b-d852-4e1a-a881-7ea8050c6562","Type":"ContainerStarted","Data":"593846b408782394542f745cbeee1bace8b6bc5fd7ea9605f5a6d3bba252b6ef"}
Jul 17 15:10:40.719014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.718895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm" event={"ID":"2b11a47b-d852-4e1a-a881-7ea8050c6562","Type":"ContainerStarted","Data":"da7dca7a6960aadd7a57ac240e67fb786b7107deb0a9c6c5d72f13f36974e067"}
Jul 17 15:10:40.719014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:40.718925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm" event={"ID":"2b11a47b-d852-4e1a-a881-7ea8050c6562","Type":"ContainerStarted","Data":"1f8ee5360cca5c7f9c11a90cb4f35a5e96687a6ac55f03e997fd1504bcfa56f9"}
Jul 17 15:10:41.022366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.022316    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:41.022694 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.022418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:41.028270 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.028245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:41.355986 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.355897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:41.358767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.358742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71506a20-0b90-4314-a750-f44ebb0a855b-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75vzx\" (UID: \"71506a20-0b90-4314-a750-f44ebb0a855b\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:41.441479 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.441443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx"
Jul 17 15:10:41.736488 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.736310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:10:41.777877 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:41.777176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75vzx"]
Jul 17 15:10:41.783145 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:41.783111    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71506a20_0b90_4314_a750_f44ebb0a855b.slice/crio-20db97eaf1416cdca052ea2c3f8e15de0a865d9d9e0c83573774d7fbd1d258d2 WatchSource:0}: Error finding container 20db97eaf1416cdca052ea2c3f8e15de0a865d9d9e0c83573774d7fbd1d258d2: Status 404 returned error can't find the container with id 20db97eaf1416cdca052ea2c3f8e15de0a865d9d9e0c83573774d7fbd1d258d2
Jul 17 15:10:42.583810 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.583767    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7778f65b86-55grl"]
Jul 17 15:10:42.608538 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.608508    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7778f65b86-55grl"]
Jul 17 15:10:42.608714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.608670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.612089 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.612041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 17 15:10:42.612609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.612582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2wbqf\""
Jul 17 15:10:42.612745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.612588    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 17 15:10:42.612745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.612652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 17 15:10:42.612745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.612595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 17 15:10:42.613083 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.613038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 17 15:10:42.613201 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.613038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8heggjekphtoi\""
Jul 17 15:10:42.733190 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.733158    2566 generic.go:356] "Generic (PLEG): container finished" podID="09d5b15f-e7f4-490a-9713-452b2ddf077c" containerID="4920f15f5267b1a4d45d763335e21993948bce31eb0905a34f2d599a22d6beda" exitCode=0
Jul 17 15:10:42.733365 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.733253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dg4kt" event={"ID":"09d5b15f-e7f4-490a-9713-452b2ddf077c","Type":"ContainerDied","Data":"4920f15f5267b1a4d45d763335e21993948bce31eb0905a34f2d599a22d6beda"}
Jul 17 15:10:42.734519 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.734498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx" event={"ID":"71506a20-0b90-4314-a750-f44ebb0a855b","Type":"ContainerStarted","Data":"20db97eaf1416cdca052ea2c3f8e15de0a865d9d9e0c83573774d7fbd1d258d2"}
Jul 17 15:10:42.768738 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-metrics-client-ca\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-grpc-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.769163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.768939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2wm\" (UniqueName: \"kubernetes.io/projected/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-kube-api-access-kk2wm\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869564 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869564 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869564 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-metrics-client-ca\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869718 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869718 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-grpc-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.869869 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.869838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2wm\" (UniqueName: \"kubernetes.io/projected/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-kube-api-access-kk2wm\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.870677 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.870644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-metrics-client-ca\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.873263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.873214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.874181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.874018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.874181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.874140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.875143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.875012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-grpc-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.876251 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.876208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.877214 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.877174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-secret-thanos-querier-tls\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.880321 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.880288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2wm\" (UniqueName: \"kubernetes.io/projected/446073c0-4064-4c2f-a1ad-bcc0c91aec4f-kube-api-access-kk2wm\") pod \"thanos-querier-7778f65b86-55grl\" (UID: \"446073c0-4064-4c2f-a1ad-bcc0c91aec4f\") " pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:42.919881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:42.919855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:43.100754 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.100727    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7778f65b86-55grl"]
Jul 17 15:10:43.102466 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:43.102438    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446073c0_4064_4c2f_a1ad_bcc0c91aec4f.slice/crio-529da0835f68d047662342d4cbf66425b3cce8352d0780c59f02feaa7436b34a WatchSource:0}: Error finding container 529da0835f68d047662342d4cbf66425b3cce8352d0780c59f02feaa7436b34a: Status 404 returned error can't find the container with id 529da0835f68d047662342d4cbf66425b3cce8352d0780c59f02feaa7436b34a
Jul 17 15:10:43.740382 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.740348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"529da0835f68d047662342d4cbf66425b3cce8352d0780c59f02feaa7436b34a"}
Jul 17 15:10:43.742454 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.742425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dg4kt" event={"ID":"09d5b15f-e7f4-490a-9713-452b2ddf077c","Type":"ContainerStarted","Data":"13f2dac7e41070d4ede0b3331973d469579abf4b6b9c12ad7ebae42844eeb547"}
Jul 17 15:10:43.742610 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.742482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dg4kt" event={"ID":"09d5b15f-e7f4-490a-9713-452b2ddf077c","Type":"ContainerStarted","Data":"d6d54111a6716820edb70dc3e7e3f1403051fe4172c4eab0cd3c9cbdc43b4d54"}
Jul 17 15:10:43.744333 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.744311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm" event={"ID":"2b11a47b-d852-4e1a-a881-7ea8050c6562","Type":"ContainerStarted","Data":"00a56021bde1cea373f351bfe1d763ebafcae955f60ce462b98b9ca4765b48a4"}
Jul 17 15:10:43.771761 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.771708    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dg4kt" podStartSLOduration=3.062768164 podStartE2EDuration="4.771690345s" totalImagesPullingTime="1.589279065s" totalInitContainerRuntime="119.643116ms" isStatefulPod=true podCreationTimestamp="2026-07-17 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:43.77038819 +0000 UTC m=+94.045536514" watchObservedRunningTime="2026-07-17 15:10:43.771690345 +0000 UTC m=+94.046838665"
Jul 17 15:10:43.795639 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.795440    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-mgffm" podStartSLOduration=2.429892769 podStartE2EDuration="4.795422094s" totalImagesPullingTime="2.365529325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:43.79541721 +0000 UTC m=+94.070565533" watchObservedRunningTime="2026-07-17 15:10:43.795422094 +0000 UTC m=+94.070570414"
Jul 17 15:10:43.912567 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.912527    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"]
Jul 17 15:10:43.930828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.930666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.933338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.933204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"]
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942127    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942142    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942154    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942132    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jvzcn\""
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942304    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 17 15:10:43.942376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.942375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fdjan0du7d9bb\""
Jul 17 15:10:43.980021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.979988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c64b493-6007-4d6f-8f38-a31c390af8a3-audit-log\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-client-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zk8m\" (UniqueName: \"kubernetes.io/projected/8c64b493-6007-4d6f-8f38-a31c390af8a3-kube-api-access-9zk8m\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-client-certs\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980357 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-metrics-server-audit-profiles\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:43.980357 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:43.980281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-tls\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081475 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-tls\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c64b493-6007-4d6f-8f38-a31c390af8a3-audit-log\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-client-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zk8m\" (UniqueName: \"kubernetes.io/projected/8c64b493-6007-4d6f-8f38-a31c390af8a3-kube-api-access-9zk8m\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-client-certs\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.081720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-metrics-server-audit-profiles\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.082030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.081935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8c64b493-6007-4d6f-8f38-a31c390af8a3-audit-log\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.082249 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.082223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.082565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.082548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8c64b493-6007-4d6f-8f38-a31c390af8a3-metrics-server-audit-profiles\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.084487 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.084455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-client-certs\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.085074 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.085033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-secret-metrics-server-tls\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.085163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.085142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8c64b493-6007-4d6f-8f38-a31c390af8a3-client-ca-bundle\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.097628 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.097561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zk8m\" (UniqueName: \"kubernetes.io/projected/8c64b493-6007-4d6f-8f38-a31c390af8a3-kube-api-access-9zk8m\") pod \"metrics-server-6bff5fbbbc-lk6jj\" (UID: \"8c64b493-6007-4d6f-8f38-a31c390af8a3\") " pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.245694 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.245626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:10:44.488436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.488404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"]
Jul 17 15:10:44.492460 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:44.492427    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c64b493_6007_4d6f_8f38_a31c390af8a3.slice/crio-3c04e762e6af9935566c1f7155ad318260967f17d1e7c41cc15800f5778702f6 WatchSource:0}: Error finding container 3c04e762e6af9935566c1f7155ad318260967f17d1e7c41cc15800f5778702f6: Status 404 returned error can't find the container with id 3c04e762e6af9935566c1f7155ad318260967f17d1e7c41cc15800f5778702f6
Jul 17 15:10:44.517265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.517228    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:10:44.534718 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.534664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:10:44.534880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.534814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586527 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7qw\" (UniqueName: \"kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.586912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.586735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688300 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688300 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7qw\" (UniqueName: \"kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.688648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.688642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.689041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.689020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.689494 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.689464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.689616 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.689601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.689791 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.689769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.691488 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.691470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.691766 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.691745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.701224 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.701202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7qw\" (UniqueName: \"kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw\") pod \"console-664b85c9fd-gg48l\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") " pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:44.751239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.751141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx" event={"ID":"71506a20-0b90-4314-a750-f44ebb0a855b","Type":"ContainerStarted","Data":"98e4061ddf6ae8ad4e183db424569f6c0be80c5c19992a9a7762e8ec2b1a493e"}
Jul 17 15:10:44.751239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.751221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx" event={"ID":"71506a20-0b90-4314-a750-f44ebb0a855b","Type":"ContainerStarted","Data":"e6b7ba95545ad48b39d429f6c204ccc809d636c16e0492e1b7873b5ead8c49bb"}
Jul 17 15:10:44.751485 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.751252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx" event={"ID":"71506a20-0b90-4314-a750-f44ebb0a855b","Type":"ContainerStarted","Data":"a031615dd671e9df062f80e0a7e5db78ffa7d5e1afef6af1bcd9c33c0ad538b2"}
Jul 17 15:10:44.752870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.752811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj" event={"ID":"8c64b493-6007-4d6f-8f38-a31c390af8a3","Type":"ContainerStarted","Data":"3c04e762e6af9935566c1f7155ad318260967f17d1e7c41cc15800f5778702f6"}
Jul 17 15:10:44.778728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.778678    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-75vzx" podStartSLOduration=3.62358844 podStartE2EDuration="5.778659727s" totalImagesPullingTime="2.155071287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:44.7779228 +0000 UTC m=+95.053071122" watchObservedRunningTime="2026-07-17 15:10:44.778659727 +0000 UTC m=+95.053808048"
Jul 17 15:10:44.847160 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:44.847123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:45.026827 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.026744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:10:45.029948 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:45.029920    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b64911_5ed4_49f7_a7fb_02183d493448.slice/crio-bf474bf6b7e966d2a37cb4143894598014807f8ccd4992b22167c31af445b9fe WatchSource:0}: Error finding container bf474bf6b7e966d2a37cb4143894598014807f8ccd4992b22167c31af445b9fe: Status 404 returned error can't find the container with id bf474bf6b7e966d2a37cb4143894598014807f8ccd4992b22167c31af445b9fe
Jul 17 15:10:45.762830 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.762358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664b85c9fd-gg48l" event={"ID":"66b64911-5ed4-49f7-a7fb-02183d493448","Type":"ContainerStarted","Data":"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"}
Jul 17 15:10:45.762830 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.762454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664b85c9fd-gg48l" event={"ID":"66b64911-5ed4-49f7-a7fb-02183d493448","Type":"ContainerStarted","Data":"bf474bf6b7e966d2a37cb4143894598014807f8ccd4992b22167c31af445b9fe"}
Jul 17 15:10:45.906464 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.906298    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-664b85c9fd-gg48l" podStartSLOduration=1.906273803 podStartE2EDuration="1.906273803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:45.784323551 +0000 UTC m=+96.059471872" watchObservedRunningTime="2026-07-17 15:10:45.906273803 +0000 UTC m=+96.181422124"
Jul 17 15:10:45.907009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.906941    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:10:45.928225 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.928191    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:45.931617 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.931589    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 15:10:45.932901 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.931616    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 15:10:45.933229 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.932011    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 15:10:45.933474 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.932115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 15:10:45.933712 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.932179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 15:10:45.933958 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.932662    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1ff7ot1fo1lb8\""
Jul 17 15:10:45.934226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.932909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 15:10:45.934693 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.934675    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.934827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.934761    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.935557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.936026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.936288    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ph2pl\""
Jul 17 15:10:45.937019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.936466    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 15:10:45.938076 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.937734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 15:10:45.938631 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.938543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 15:10:45.948982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:45.948956    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 15:10:46.002079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002265 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002536 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrfl\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.002922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.002698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103568 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103568 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103568 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrfl\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.103847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.103687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.104510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.104510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.104924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.106246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.105803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.108397 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.108363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.108572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.108553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.109288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.109149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.110770 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.110305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.110770 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.110501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.110770 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.110730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.111174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.110911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.111174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.111128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.111309 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.111291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.111887 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.111847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.111983 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.111943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113577 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.113671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.113639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.119425 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.119401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrfl\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl\") pod \"prometheus-k8s-0\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.245664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.245634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:10:46.955459 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:46.955436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:10:46.958796 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:10:46.958762    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229865cc_e40f_4a30_a476_32fb1e9d4027.slice/crio-081497c18584db5057e75ecf51aad1b88a4e376a97bafd56774f4c3e1b9de64d WatchSource:0}: Error finding container 081497c18584db5057e75ecf51aad1b88a4e376a97bafd56774f4c3e1b9de64d: Status 404 returned error can't find the container with id 081497c18584db5057e75ecf51aad1b88a4e376a97bafd56774f4c3e1b9de64d
Jul 17 15:10:47.772005 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.771965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"86f853323616e43f8543d69d2359ec15105df28bceb6a150a2ac3d8ef92461bd"}
Jul 17 15:10:47.772258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.772109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"bf1fe83ecbe104ab8f992549ad8c9ed0d55f53fb73e090df22f629fc292c50eb"}
Jul 17 15:10:47.772258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.772159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"a1b497761980ccd4fd18ae43dd55935213e670fcc84918005a0ad1faadab149d"}
Jul 17 15:10:47.773316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.773289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"081497c18584db5057e75ecf51aad1b88a4e376a97bafd56774f4c3e1b9de64d"}
Jul 17 15:10:47.774925 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.774896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj" event={"ID":"8c64b493-6007-4d6f-8f38-a31c390af8a3","Type":"ContainerStarted","Data":"480d4883eba76fe975822e9c64f681cafd31424cc08d8d1684154bfb502521c8"}
Jul 17 15:10:47.810743 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:47.810694    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj" podStartSLOduration=2.501695122 podStartE2EDuration="4.81067585s" totalImagesPullingTime="2.308980728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:47.808793757 +0000 UTC m=+98.083942079" watchObservedRunningTime="2026-07-17 15:10:47.81067585 +0000 UTC m=+98.085824172"
Jul 17 15:10:48.782161 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:48.782122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"}
Jul 17 15:10:49.790299 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.790267    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945" exitCode=0
Jul 17 15:10:49.790879 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.790360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"}
Jul 17 15:10:49.793601 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.793580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"025cb9eb5e5c83fb17a29c5635025aaae813107ef17f75b351a07231c22b6250"}
Jul 17 15:10:49.793667 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.793655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"11f110331dbdd778255d7d9ab613f2d73b60424ff21354f681a2510fd6302eec"}
Jul 17 15:10:49.793706 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.793695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" event={"ID":"446073c0-4064-4c2f-a1ad-bcc0c91aec4f","Type":"ContainerStarted","Data":"e00ed9a08d39ecc516ca1b7b253feb7a7175669c213b58810d2b7498e9dc1ec9"}
Jul 17 15:10:49.793826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.793807    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:49.852122 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:49.852070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl" podStartSLOduration=2.654928267 podStartE2EDuration="7.852038361s" totalImagesPullingTime="5.197110094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:49.850893722 +0000 UTC m=+100.126042062" watchObservedRunningTime="2026-07-17 15:10:49.852038361 +0000 UTC m=+100.127186682"
Jul 17 15:10:53.819143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:53.818972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"}
Jul 17 15:10:53.819143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:53.819077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"}
Jul 17 15:10:54.827913 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.827868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"}
Jul 17 15:10:54.828403 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.827943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"}
Jul 17 15:10:54.828403 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.827982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"}
Jul 17 15:10:54.828403 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.828010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerStarted","Data":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"}
Jul 17 15:10:54.847718 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.847677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:54.847874 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.847740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:54.852634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.852603    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:54.860717 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:54.860660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.403328984 podStartE2EDuration="9.860629918s" totalImagesPullingTime="5.436662903s" totalInitContainerRuntime="20.638031ms" isStatefulPod=false podCreationTimestamp="2026-07-17 15:10:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:10:54.857874556 +0000 UTC m=+105.133022877" watchObservedRunningTime="2026-07-17 15:10:54.860629918 +0000 UTC m=+105.135778241"
Jul 17 15:10:55.804680 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:55.804654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7778f65b86-55grl"
Jul 17 15:10:55.840678 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:55.840639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:10:55.893945 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:55.893913    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:10:56.246027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:10:56.245992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:11:04.246788 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:04.246750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:11:04.247308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:04.246810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:11:04.674926 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:04.674828    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8v2c8"
Jul 17 15:11:20.927021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:20.926973    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d9798548f-jjk45" podUID="73c9987d-79b7-47ab-857d-3f9d490f998a" containerName="console" containerID="cri-o://169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d" gracePeriod=15
Jul 17 15:11:21.206149 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.206044    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:11:21.324433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324392    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.324623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324471    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.324623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.324623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkls2\" (UniqueName: \"kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.324822 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.324931 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.325052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.325040    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config\") pod \"73c9987d-79b7-47ab-857d-3f9d490f998a\" (UID: \"73c9987d-79b7-47ab-857d-3f9d490f998a\") "
Jul 17 15:11:21.325796 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:11:21.325936 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.324967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:11:21.326018 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.325377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca" (OuterVolumeSpecName: "service-ca") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:11:21.326018 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.325342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config" (OuterVolumeSpecName: "console-config") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:11:21.327283 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.327254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:11:21.327425 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.327403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2" (OuterVolumeSpecName: "kube-api-access-zkls2") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "kube-api-access-zkls2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:11:21.327827 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.327809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73c9987d-79b7-47ab-857d-3f9d490f998a" (UID: "73c9987d-79b7-47ab-857d-3f9d490f998a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:11:21.426756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426720    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-oauth-serving-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.426756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426753    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-trusted-ca-bundle\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.426756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426765    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-serving-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.427103 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426774    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkls2\" (UniqueName: \"kubernetes.io/projected/73c9987d-79b7-47ab-857d-3f9d490f998a-kube-api-access-zkls2\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.427103 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426784    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-console-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.427103 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426796    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73c9987d-79b7-47ab-857d-3f9d490f998a-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.427103 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.426807    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73c9987d-79b7-47ab-857d-3f9d490f998a-console-oauth-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:11:21.942155 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.942112    2566 generic.go:356] "Generic (PLEG): container finished" podID="73c9987d-79b7-47ab-857d-3f9d490f998a" containerID="169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d" exitCode=0
Jul 17 15:11:21.942529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.942169    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9798548f-jjk45"
Jul 17 15:11:21.942529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.942209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9798548f-jjk45" event={"ID":"73c9987d-79b7-47ab-857d-3f9d490f998a","Type":"ContainerDied","Data":"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"}
Jul 17 15:11:21.942529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.942281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9798548f-jjk45" event={"ID":"73c9987d-79b7-47ab-857d-3f9d490f998a","Type":"ContainerDied","Data":"ac3c59a0f01d90b006ab191c04e97b07c48415f3e5a8a59cff4797ff56cc0a4f"}
Jul 17 15:11:21.942529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.942301    2566 scope.go:117] "RemoveContainer" containerID="169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"
Jul 17 15:11:21.950883 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.950864    2566 scope.go:117] "RemoveContainer" containerID="169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"
Jul 17 15:11:21.951171 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:11:21.951150    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d\": container with ID starting with 169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d not found: ID does not exist" containerID="169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"
Jul 17 15:11:21.951246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.951183    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d"} err="failed to get container status \"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d\": rpc error: code = NotFound desc = could not find container \"169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d\": container with ID starting with 169ce6a2918a3ed934bdd49124ce9048db4cb8537fcdc3da93b744f8d4f0274d not found: ID does not exist"
Jul 17 15:11:21.976378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.976340    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:11:21.980698 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:21.980672    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9798548f-jjk45"]
Jul 17 15:11:22.240128 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:22.240101    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c9987d-79b7-47ab-857d-3f9d490f998a" path="/var/lib/kubelet/pods/73c9987d-79b7-47ab-857d-3f9d490f998a/volumes"
Jul 17 15:11:24.251733 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:24.251706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:11:24.255711 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:24.255685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bff5fbbbc-lk6jj"
Jul 17 15:11:46.246892 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:46.246858    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:11:46.267043 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:46.267014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:11:47.058689 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:11:47.058661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:04.513547 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513442    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:04.514027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513840    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy" containerID="cri-o://b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04" gracePeriod=600
Jul 17 15:12:04.514027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513887    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-web" containerID="cri-o://a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c" gracePeriod=600
Jul 17 15:12:04.514255 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513905    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-thanos" containerID="cri-o://89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9" gracePeriod=600
Jul 17 15:12:04.514255 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513905    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="thanos-sidecar" containerID="cri-o://3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869" gracePeriod=600
Jul 17 15:12:04.514255 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513840    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="prometheus" containerID="cri-o://73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6" gracePeriod=600
Jul 17 15:12:04.514407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.513957    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="config-reloader" containerID="cri-o://1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003" gracePeriod=600
Jul 17 15:12:04.773245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.772849    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:04.924812 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.924812 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924813    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wrfl\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924892    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924919    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924935    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.924976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925009    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925094    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925149    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925296    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925369    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925408    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.925480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls\") pod \"229865cc-e40f-4a30-a476-32fb1e9d4027\" (UID: \"229865cc-e40f-4a30-a476-32fb1e9d4027\") "
Jul 17 15:12:04.926217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.925895    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:04.926273 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.926216    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.926273 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.926243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.926534 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.926504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.927285 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.927253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.927628 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.927604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.928002 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.927967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.928401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.928359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.928512 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.928433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config" (OuterVolumeSpecName: "config") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.928597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.928514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.928597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.928511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:04.928597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.928583    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl" (OuterVolumeSpecName: "kube-api-access-5wrfl") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "kube-api-access-5wrfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:12:04.929660 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.929634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:12:04.930534 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.930624 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.930624 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out" (OuterVolumeSpecName: "config-out") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:12:04.930624 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930612    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:12:04.930782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.930910 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.930890    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:04.941125 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:04.941099    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config" (OuterVolumeSpecName: "web-config") pod "229865cc-e40f-4a30-a476-32fb1e9d4027" (UID: "229865cc-e40f-4a30-a476-32fb1e9d4027"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026323    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026362    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026376    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026389    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026401    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-metrics-client-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026415    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026428    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-tls-assets\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026441    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-db\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026453    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-metrics-client-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026464    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/229865cc-e40f-4a30-a476-32fb1e9d4027-config-out\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026475    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026488    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026500    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-kube-rbac-proxy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026512    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/229865cc-e40f-4a30-a476-32fb1e9d4027-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026524    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026536    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-secret-grpc-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026547    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrfl\" (UniqueName: \"kubernetes.io/projected/229865cc-e40f-4a30-a476-32fb1e9d4027-kube-api-access-5wrfl\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026560    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.026789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.026571    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/229865cc-e40f-4a30-a476-32fb1e9d4027-web-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:05.120934 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120901    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9" exitCode=0
Jul 17 15:12:05.120934 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120928    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04" exitCode=0
Jul 17 15:12:05.120934 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120939    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c" exitCode=0
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120950    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869" exitCode=0
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120960    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003" exitCode=0
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120967    2566 generic.go:356] "Generic (PLEG): container finished" podID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6" exitCode=0
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"}
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.120981    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"}
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121127    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.121239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"}
Jul 17 15:12:05.121591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"}
Jul 17 15:12:05.121591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"}
Jul 17 15:12:05.121591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"}
Jul 17 15:12:05.121591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.121431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"229865cc-e40f-4a30-a476-32fb1e9d4027","Type":"ContainerDied","Data":"081497c18584db5057e75ecf51aad1b88a4e376a97bafd56774f4c3e1b9de64d"}
Jul 17 15:12:05.129644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.129625    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.137525 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.137505    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.145247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.145158    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.146573 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.146543    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:05.152900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.152845    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:05.154195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.154177    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.161954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.161935    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.169917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.169895    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.177565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.177545    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.177883 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.177862    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.177970 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.177888    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.177970 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.177908    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.178230 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.178211    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.178279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178236    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.178279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178254    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.178484 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.178470    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.178521 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178488    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.178521 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178502    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.178708 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.178694    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.178744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178711    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.178744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178726    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.178934 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.178916    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.178982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178944    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.178982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.178960    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.179236 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.179221    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.179295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179240    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.179295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179252    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.179485 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:05.179466    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.179604 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179502    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.179604 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179527    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.179766 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179749    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.179829 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.179768    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.180047 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180030    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.180112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180048    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.180286 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180271    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.180330 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180287    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.180493 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180479    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.180542 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180493    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.180683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180667    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.180683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180683    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.180867 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180850    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.180935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.180869    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.181143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181125    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.181143 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181143    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.181372 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181354    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.181445 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181374    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.181587 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.181634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181588    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.181811 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181791    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.181875 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.181812    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.182181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182106    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.182181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182129    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.182639 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182617    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.182639 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182639    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.183139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182908    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.183139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.182928    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.183369 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183348    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.183369 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183369    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.183596 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183580    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.183596 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183596    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.183823 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183804    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.183868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.183823    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.184044 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184024    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.184044 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:05.184195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184046    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.184367 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184351    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.184414 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184367    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.184414 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184405    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-web"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184417    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-web"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184436    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="init-config-reloader"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184442    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="init-config-reloader"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184455    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184464    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184473    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="config-reloader"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184481    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="config-reloader"
Jul 17 15:12:05.184491 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184491    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-thanos"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184499    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-thanos"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184512    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c9987d-79b7-47ab-857d-3f9d490f998a" containerName="console"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184520    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c9987d-79b7-47ab-857d-3f9d490f998a" containerName="console"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184527    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="thanos-sidecar"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184532    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="thanos-sidecar"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184540    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="prometheus"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184545    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="prometheus"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184575    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184588    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184590    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-web"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184635    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184648    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="config-reloader"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184656    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c9987d-79b7-47ab-857d-3f9d490f998a" containerName="console"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184662    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="thanos-sidecar"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184668    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="prometheus"
Jul 17 15:12:05.184752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184673    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" containerName="kube-rbac-proxy-thanos"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184788    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.184805    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185007    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185030    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185277    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.185363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185303    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.185637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185494    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.185637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185513    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.185728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185711    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.185772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185730    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.185903 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185887    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.185952 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.185903    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.186132 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186113    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.186185 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186132    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.186351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186334    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.186412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186353    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.186593 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186576    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.186642 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186595    2566 scope.go:117] "RemoveContainer" containerID="89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"
Jul 17 15:12:05.186781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186764    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9"} err="failed to get container status \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": rpc error: code = NotFound desc = could not find container \"89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9\": container with ID starting with 89927f59ed0acc7edd9868e81c3a69004b49ffda0aecf632b719f9650a587de9 not found: ID does not exist"
Jul 17 15:12:05.186847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186783    2566 scope.go:117] "RemoveContainer" containerID="b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"
Jul 17 15:12:05.187010 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.186993    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04"} err="failed to get container status \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": rpc error: code = NotFound desc = could not find container \"b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04\": container with ID starting with b9dd250ffe09eded79a7643b8df571f597ca3bb3cd4ced4c5779af132d85fa04 not found: ID does not exist"
Jul 17 15:12:05.187069 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.187010    2566 scope.go:117] "RemoveContainer" containerID="a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"
Jul 17 15:12:05.187217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.187202    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c"} err="failed to get container status \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": rpc error: code = NotFound desc = could not find container \"a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c\": container with ID starting with a271d224fbd189c6d606f07f5943b15ae6526997f8f79706662cf77f525c6c4c not found: ID does not exist"
Jul 17 15:12:05.187217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.187217    2566 scope.go:117] "RemoveContainer" containerID="3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"
Jul 17 15:12:05.187411 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.187393    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869"} err="failed to get container status \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": rpc error: code = NotFound desc = could not find container \"3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869\": container with ID starting with 3807717510c25581d89c89a18d86005953d173842fb55f5bd7d852ced455d869 not found: ID does not exist"
Jul 17 15:12:05.187480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.187412    2566 scope.go:117] "RemoveContainer" containerID="1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"
Jul 17 15:12:05.188163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188146    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003"} err="failed to get container status \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": rpc error: code = NotFound desc = could not find container \"1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003\": container with ID starting with 1c312e143051ea124af43f6f478c8e83391a4f69278f68e7aa5172f97703c003 not found: ID does not exist"
Jul 17 15:12:05.188244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188166    2566 scope.go:117] "RemoveContainer" containerID="73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"
Jul 17 15:12:05.188244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.188632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188420    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6"} err="failed to get container status \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": rpc error: code = NotFound desc = could not find container \"73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6\": container with ID starting with 73c74183cbff52ed61cb79269b9b7dc21d9ad54eae470a0cf28079def70e4da6 not found: ID does not exist"
Jul 17 15:12:05.188632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188444    2566 scope.go:117] "RemoveContainer" containerID="1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"
Jul 17 15:12:05.188878 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.188842    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945"} err="failed to get container status \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": rpc error: code = NotFound desc = could not find container \"1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945\": container with ID starting with 1f3fc26de769fdbdb1b72945e12c73fbbb07a6ee470129ef0466a3ab21798945 not found: ID does not exist"
Jul 17 15:12:05.190742 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.190722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 15:12:05.190858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.190776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 15:12:05.190858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.190797    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 15:12:05.191002 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.190856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 15:12:05.191002 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.190872    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 15:12:05.191153 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.191139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 15:12:05.191768 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.191532    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 15:12:05.191768 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.191570    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 15:12:05.191768 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.191749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 15:12:05.192076 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.192038    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 15:12:05.192167 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.192148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 15:12:05.192219 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.192043    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 15:12:05.192392 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.192376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1ff7ot1fo1lb8\""
Jul 17 15:12:05.192460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.192410    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ph2pl\""
Jul 17 15:12:05.195151 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.195131    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 15:12:05.197441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.197423    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 15:12:05.208170 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.208138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:05.329152 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329152 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329152 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcx4\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-kube-api-access-qxcx4\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-web-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-config-out\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.329664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.329610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.430881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.430830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.430881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.430880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.430926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.430958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.430987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-web-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-config-out\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.431997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.431951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.432163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.432005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.432163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.432083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.432163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.432118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcx4\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-kube-api-access-qxcx4\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.433736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.432691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.433736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.432946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.433736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.433382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.433736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.433612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.434340 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.434316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.434438 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.434381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.434438 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.434404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.435115 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.435090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.435653 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.435368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/29653e64-77b4-43fe-94bc-2d6d951d13da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.435804 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.435785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-web-config\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.436206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.436183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.436479 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.436459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.436832 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.436814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29653e64-77b4-43fe-94bc-2d6d951d13da-config-out\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.437155 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.437138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.437395 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.437377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.437471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.437453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/29653e64-77b4-43fe-94bc-2d6d951d13da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.441756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.441738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcx4\" (UniqueName: \"kubernetes.io/projected/29653e64-77b4-43fe-94bc-2d6d951d13da-kube-api-access-qxcx4\") pod \"prometheus-k8s-0\" (UID: \"29653e64-77b4-43fe-94bc-2d6d951d13da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.498501 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.498462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:05.650095 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:05.646982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 15:12:06.127716 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:06.127684    2566 generic.go:356] "Generic (PLEG): container finished" podID="29653e64-77b4-43fe-94bc-2d6d951d13da" containerID="09ec2736b394043cb1f58493952a940e72668bf9bad7d186f0045d91b363027b" exitCode=0
Jul 17 15:12:06.127862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:06.127748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerDied","Data":"09ec2736b394043cb1f58493952a940e72668bf9bad7d186f0045d91b363027b"}
Jul 17 15:12:06.127862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:06.127799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"950cc6d15e8e925276589d0906e48f19c66215c8b81e839951724759c602037a"}
Jul 17 15:12:06.241815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:06.241780    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229865cc-e40f-4a30-a476-32fb1e9d4027" path="/var/lib/kubelet/pods/229865cc-e40f-4a30-a476-32fb1e9d4027/volumes"
Jul 17 15:12:07.134960 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.134922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"610d5be802a583ed5147b99d15c4379fd4c99da5b580a13ac6641377f960303c"}
Jul 17 15:12:07.135387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.135011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"427eadd21a8fbd67a118db7ac5cd2217e440732243e5f99eade9703ad310edfc"}
Jul 17 15:12:07.135387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.135085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"e79793af1a86d09f6c4a3f3f8071a1bfaeee31a7dba6fa15dbea47ed453d135c"}
Jul 17 15:12:07.135387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.135122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"92e97ea8aa0effb58ec7faf98c32cd2b58fd90b16a7a5832701b4a10203788b2"}
Jul 17 15:12:07.135387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.135167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"f384e70cffbb3985a37480f061989459fb06e0b3ed3fa8785d46b3e833844410"}
Jul 17 15:12:07.135387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.135204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"29653e64-77b4-43fe-94bc-2d6d951d13da","Type":"ContainerStarted","Data":"0b20945f3928da3be909c7a01bf2f8bbff1b1a4aa84a8d7de9f2b5e411da7f43"}
Jul 17 15:12:07.170833 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:07.170776    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.147293302 podStartE2EDuration="2.170756685s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.463383ms" isStatefulPod=false podCreationTimestamp="2026-07-17 15:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:12:07.169684546 +0000 UTC m=+177.444832888" watchObservedRunningTime="2026-07-17 15:12:07.170756685 +0000 UTC m=+177.445905006"
Jul 17 15:12:10.499209 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:10.499179    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:12:19.495772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:19.495733    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:12:44.516766 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.516702    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-664b85c9fd-gg48l" podUID="66b64911-5ed4-49f7-a7fb-02183d493448" containerName="console" containerID="cri-o://b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74" gracePeriod=15
Jul 17 15:12:44.763079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.763041    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:12:44.879404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w7qw\" (UniqueName: \"kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config\") pod \"66b64911-5ed4-49f7-a7fb-02183d493448\" (UID: \"66b64911-5ed4-49f7-a7fb-02183d493448\") "
Jul 17 15:12:44.879889 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:44.879889 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879852    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:44.879989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config" (OuterVolumeSpecName: "console-config") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:44.879989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.879904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca" (OuterVolumeSpecName: "service-ca") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:12:44.881755 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.881725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:44.881876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.881788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:12:44.881876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.881806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw" (OuterVolumeSpecName: "kube-api-access-8w7qw") pod "66b64911-5ed4-49f7-a7fb-02183d493448" (UID: "66b64911-5ed4-49f7-a7fb-02183d493448"). InnerVolumeSpecName "kube-api-access-8w7qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:12:44.981044 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981011    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-trusted-ca-bundle\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981044 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981040    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-serving-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981252 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981050    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-console-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981252 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981081    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w7qw\" (UniqueName: \"kubernetes.io/projected/66b64911-5ed4-49f7-a7fb-02183d493448-kube-api-access-8w7qw\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981252 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981090    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-oauth-serving-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981252 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981099    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66b64911-5ed4-49f7-a7fb-02183d493448-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:44.981252 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:44.981106    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66b64911-5ed4-49f7-a7fb-02183d493448-console-oauth-config\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:12:45.289490 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.289447    2566 generic.go:356] "Generic (PLEG): container finished" podID="66b64911-5ed4-49f7-a7fb-02183d493448" containerID="b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74" exitCode=0
Jul 17 15:12:45.289676 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.289495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664b85c9fd-gg48l" event={"ID":"66b64911-5ed4-49f7-a7fb-02183d493448","Type":"ContainerDied","Data":"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"}
Jul 17 15:12:45.289676 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.289509    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664b85c9fd-gg48l"
Jul 17 15:12:45.289676 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.289554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664b85c9fd-gg48l" event={"ID":"66b64911-5ed4-49f7-a7fb-02183d493448","Type":"ContainerDied","Data":"bf474bf6b7e966d2a37cb4143894598014807f8ccd4992b22167c31af445b9fe"}
Jul 17 15:12:45.289676 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.289582    2566 scope.go:117] "RemoveContainer" containerID="b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"
Jul 17 15:12:45.303104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.303077    2566 scope.go:117] "RemoveContainer" containerID="b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"
Jul 17 15:12:45.303452 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:12:45.303432    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74\": container with ID starting with b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74 not found: ID does not exist" containerID="b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"
Jul 17 15:12:45.303507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.303463    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74"} err="failed to get container status \"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74\": rpc error: code = NotFound desc = could not find container \"b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74\": container with ID starting with b7fecdc592f04249770e4e96a7cce6e438e0f33db319f7d038d52f73a69d1c74 not found: ID does not exist"
Jul 17 15:12:45.316994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.316969    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:12:45.325392 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:45.325367    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-664b85c9fd-gg48l"]
Jul 17 15:12:46.240026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:12:46.239994    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b64911-5ed4-49f7-a7fb-02183d493448" path="/var/lib/kubelet/pods/66b64911-5ed4-49f7-a7fb-02183d493448/volumes"
Jul 17 15:13:05.499463 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:13:05.499413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:13:05.515484 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:13:05.515456    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:13:06.385845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:13:06.385817    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 15:14:10.109269 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:14:10.109239    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:14:10.110172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:14:10.110148    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:14:10.116443 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:14:10.116422    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 15:15:44.606917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.606883    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"]
Jul 17 15:15:44.607421 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.607194    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b64911-5ed4-49f7-a7fb-02183d493448" containerName="console"
Jul 17 15:15:44.607421 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.607204    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b64911-5ed4-49f7-a7fb-02183d493448" containerName="console"
Jul 17 15:15:44.607421 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.607272    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b64911-5ed4-49f7-a7fb-02183d493448" containerName="console"
Jul 17 15:15:44.609070 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.609043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.611406 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.611374    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qzq56\""
Jul 17 15:15:44.611565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.611434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 15:15:44.611565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.611384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 15:15:44.611565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.611464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 15:15:44.611803 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.611786    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 15:15:44.627492 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.627455    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"]
Jul 17 15:15:44.688498 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.688457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-manifests\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.688498 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.688496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.688702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.688526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-tmp\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.688702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.688605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-kube-api-access-z2tnd\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.688702 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.688637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.789848 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.789811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.789994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.789867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-tmp\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.789994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.789905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-kube-api-access-z2tnd\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.789994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.789928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.789994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.789972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-manifests\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.790460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.790429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-manifests\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.792378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.792355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-tmp\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.792495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.792448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.792563 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.792524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.801716 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.801693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tnd\" (UniqueName: \"kubernetes.io/projected/734db4ee-d1c7-4e56-aa75-e16ee5168fa1-kube-api-access-z2tnd\") pod \"opendatahub-operator-controller-manager-6fb95c7566-zxvfw\" (UID: \"734db4ee-d1c7-4e56-aa75-e16ee5168fa1\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:44.919805 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:44.919715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:45.069522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:45.069483    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"]
Jul 17 15:15:45.074919 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:15:45.074892    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734db4ee_d1c7_4e56_aa75_e16ee5168fa1.slice/crio-3c53dfe9a246dde66fbae016cc0077ef266d7ecde90d442120d9bc4e8ec20532 WatchSource:0}: Error finding container 3c53dfe9a246dde66fbae016cc0077ef266d7ecde90d442120d9bc4e8ec20532: Status 404 returned error can't find the container with id 3c53dfe9a246dde66fbae016cc0077ef266d7ecde90d442120d9bc4e8ec20532
Jul 17 15:15:45.076719 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:45.076699    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:15:45.971937 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:45.971854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw" event={"ID":"734db4ee-d1c7-4e56-aa75-e16ee5168fa1","Type":"ContainerStarted","Data":"3c53dfe9a246dde66fbae016cc0077ef266d7ecde90d442120d9bc4e8ec20532"}
Jul 17 15:15:48.986597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:48.986563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw" event={"ID":"734db4ee-d1c7-4e56-aa75-e16ee5168fa1","Type":"ContainerStarted","Data":"92c6022ec5044066fbb493dea80e0acdc5439e878d6d817a532c4b87c4fab0ed"}
Jul 17 15:15:49.991644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:49.991610    2566 generic.go:356] "Generic (PLEG): container finished" podID="734db4ee-d1c7-4e56-aa75-e16ee5168fa1" containerID="92c6022ec5044066fbb493dea80e0acdc5439e878d6d817a532c4b87c4fab0ed" exitCode=0
Jul 17 15:15:49.992088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:49.991704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw" event={"ID":"734db4ee-d1c7-4e56-aa75-e16ee5168fa1","Type":"ContainerDied","Data":"92c6022ec5044066fbb493dea80e0acdc5439e878d6d817a532c4b87c4fab0ed"}
Jul 17 15:15:50.997900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:50.997866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw" event={"ID":"734db4ee-d1c7-4e56-aa75-e16ee5168fa1","Type":"ContainerStarted","Data":"7f3d8b23477d5a96d34a20f043fc77e0da0b99a82b3f2950dd0854d729771466"}
Jul 17 15:15:50.998323 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:50.997967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:15:51.019850 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:15:51.019798    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw" podStartSLOduration=2.708946832 podStartE2EDuration="7.019776322s" totalImagesPullingTime="3.282747039s" totalInitContainerRuntime="1.028082451s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:15:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:15:51.018279724 +0000 UTC m=+401.293428065" watchObservedRunningTime="2026-07-17 15:15:51.019776322 +0000 UTC m=+401.294924643"
Jul 17 15:16:02.005384 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.005352    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb95c7566-zxvfw"
Jul 17 15:16:02.399762 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.399682    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"]
Jul 17 15:16:02.405169 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.405144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.407559 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407530    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 15:16:02.407674 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407560    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 15:16:02.407728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407711    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 15:16:02.407728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x8knc\""
Jul 17 15:16:02.407822 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 15:16:02.407876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.407843    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 15:16:02.413947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.413921    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"]
Jul 17 15:16:02.447973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.447935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec5e6ff9-705f-421c-b191-eae8ece4826c-manager-config\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.448196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.447995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldth\" (UniqueName: \"kubernetes.io/projected/ec5e6ff9-705f-421c-b191-eae8ece4826c-kube-api-access-lldth\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.448196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.448030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.448196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.448116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-metrics-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.548879 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.548840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldth\" (UniqueName: \"kubernetes.io/projected/ec5e6ff9-705f-421c-b191-eae8ece4826c-kube-api-access-lldth\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.548879 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.548883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.549173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.548923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-metrics-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.549173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.548993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec5e6ff9-705f-421c-b191-eae8ece4826c-manager-config\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.549750 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.549724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec5e6ff9-705f-421c-b191-eae8ece4826c-manager-config\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.551861 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.551832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-metrics-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.551935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.551832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec5e6ff9-705f-421c-b191-eae8ece4826c-cert\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.557621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.557598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldth\" (UniqueName: \"kubernetes.io/projected/ec5e6ff9-705f-421c-b191-eae8ece4826c-kube-api-access-lldth\") pod \"lws-controller-manager-76c68485bc-sbqdb\" (UID: \"ec5e6ff9-705f-421c-b191-eae8ece4826c\") " pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.715884 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.715786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:02.851218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:02.851176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"]
Jul 17 15:16:02.854118 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:16:02.854090    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5e6ff9_705f_421c_b191_eae8ece4826c.slice/crio-a1f0c7442b0c3143f70c7bb6db855b7c2c0c9baccecb23766f06fc50c892ce7f WatchSource:0}: Error finding container a1f0c7442b0c3143f70c7bb6db855b7c2c0c9baccecb23766f06fc50c892ce7f: Status 404 returned error can't find the container with id a1f0c7442b0c3143f70c7bb6db855b7c2c0c9baccecb23766f06fc50c892ce7f
Jul 17 15:16:03.049145 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:03.049113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb" event={"ID":"ec5e6ff9-705f-421c-b191-eae8ece4826c","Type":"ContainerStarted","Data":"a1f0c7442b0c3143f70c7bb6db855b7c2c0c9baccecb23766f06fc50c892ce7f"}
Jul 17 15:16:06.066785 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:06.066746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb" event={"ID":"ec5e6ff9-705f-421c-b191-eae8ece4826c","Type":"ContainerStarted","Data":"f187f53f1323d69e04b007d55f196daece54e34bf9ae7479cbd6708804ba2492"}
Jul 17 15:16:06.067282 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:06.066890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:06.083888 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:06.083840    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb" podStartSLOduration=1.7074158640000001 podStartE2EDuration="4.083825856s" totalImagesPullingTime="2.376409992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:16:06.082801072 +0000 UTC m=+416.357949397" watchObservedRunningTime="2026-07-17 15:16:06.083825856 +0000 UTC m=+416.358974178"
Jul 17 15:16:15.061115 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.061079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"]
Jul 17 15:16:15.065342 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.065318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.067543 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.067507    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 15:16:15.067697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.067649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 15:16:15.067697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.067675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 15:16:15.068234 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.068211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m87gx\""
Jul 17 15:16:15.068321 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.068212    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 15:16:15.079210 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.079182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"]
Jul 17 15:16:15.160324 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.160286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.160324 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.160327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghc2h\" (UniqueName: \"kubernetes.io/projected/0db9cbb1-715b-49a2-b994-76bd4b28959d-kube-api-access-ghc2h\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.160534 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.160349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0db9cbb1-715b-49a2-b994-76bd4b28959d-tmp\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.261859 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.261819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.261859 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.261863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc2h\" (UniqueName: \"kubernetes.io/projected/0db9cbb1-715b-49a2-b994-76bd4b28959d-kube-api-access-ghc2h\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.262148 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.261894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0db9cbb1-715b-49a2-b994-76bd4b28959d-tmp\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.262148 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:16:15.262003    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 17 15:16:15.262148 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:16:15.262107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs podName:0db9cbb1-715b-49a2-b994-76bd4b28959d nodeName:}" failed. No retries permitted until 2026-07-17 15:16:15.762086201 +0000 UTC m=+426.037234506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs") pod "kube-auth-proxy-77c457bb55-wq4lj" (UID: "0db9cbb1-715b-49a2-b994-76bd4b28959d") : secret "kube-auth-proxy-tls" not found
Jul 17 15:16:15.264336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.264314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0db9cbb1-715b-49a2-b994-76bd4b28959d-tmp\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.276689 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.276667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc2h\" (UniqueName: \"kubernetes.io/projected/0db9cbb1-715b-49a2-b994-76bd4b28959d-kube-api-access-ghc2h\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.765488 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:15.765457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:15.765743 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:16:15.765587    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 17 15:16:15.765743 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:16:15.765639    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs podName:0db9cbb1-715b-49a2-b994-76bd4b28959d nodeName:}" failed. No retries permitted until 2026-07-17 15:16:16.765626003 +0000 UTC m=+427.040774302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs") pod "kube-auth-proxy-77c457bb55-wq4lj" (UID: "0db9cbb1-715b-49a2-b994-76bd4b28959d") : secret "kube-auth-proxy-tls" not found
Jul 17 15:16:16.773911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:16.773390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:16.777262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:16.777234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0db9cbb1-715b-49a2-b994-76bd4b28959d-tls-certs\") pod \"kube-auth-proxy-77c457bb55-wq4lj\" (UID: \"0db9cbb1-715b-49a2-b994-76bd4b28959d\") " pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:16.876460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:16.876419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"
Jul 17 15:16:17.035423 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:17.035396    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj"]
Jul 17 15:16:17.035767 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:16:17.035744    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db9cbb1_715b_49a2_b994_76bd4b28959d.slice/crio-6f4b251a115459c4d1ed30e59ded7722b2ed0530073ebf0e47a9a5facc58c2ba WatchSource:0}: Error finding container 6f4b251a115459c4d1ed30e59ded7722b2ed0530073ebf0e47a9a5facc58c2ba: Status 404 returned error can't find the container with id 6f4b251a115459c4d1ed30e59ded7722b2ed0530073ebf0e47a9a5facc58c2ba
Jul 17 15:16:17.073429 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:17.073400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76c68485bc-sbqdb"
Jul 17 15:16:17.117319 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:17.117276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj" event={"ID":"0db9cbb1-715b-49a2-b994-76bd4b28959d","Type":"ContainerStarted","Data":"6f4b251a115459c4d1ed30e59ded7722b2ed0530073ebf0e47a9a5facc58c2ba"}
Jul 17 15:16:21.139853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:21.139815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj" event={"ID":"0db9cbb1-715b-49a2-b994-76bd4b28959d","Type":"ContainerStarted","Data":"d25cac752aab1f28e084bb10de83667430b286850bf9bf759e7fe3de566d28fd"}
Jul 17 15:16:21.160034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:16:21.159978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77c457bb55-wq4lj" podStartSLOduration=2.752071414 podStartE2EDuration="6.159962051s" totalImagesPullingTime="3.407890637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:16:21.158516942 +0000 UTC m=+431.433665268" watchObservedRunningTime="2026-07-17 15:16:21.159962051 +0000 UTC m=+431.435110413"
Jul 17 15:17:45.933914 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.933872    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"]
Jul 17 15:17:45.937392 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.937375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:45.940032 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.940010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lh74d\""
Jul 17 15:17:45.940281 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.940266    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 15:17:45.940341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.940277    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 15:17:45.948945 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:45.948924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"]
Jul 17 15:17:46.063285 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.063242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvskl\" (UniqueName: \"kubernetes.io/projected/ab3e808c-304a-4461-8a64-64323dc1042c-kube-api-access-cvskl\") pod \"authorino-operator-75cc8d5cb7-sv9jd\" (UID: \"ab3e808c-304a-4461-8a64-64323dc1042c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:46.163740 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.163694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvskl\" (UniqueName: \"kubernetes.io/projected/ab3e808c-304a-4461-8a64-64323dc1042c-kube-api-access-cvskl\") pod \"authorino-operator-75cc8d5cb7-sv9jd\" (UID: \"ab3e808c-304a-4461-8a64-64323dc1042c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:46.174726 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.174690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvskl\" (UniqueName: \"kubernetes.io/projected/ab3e808c-304a-4461-8a64-64323dc1042c-kube-api-access-cvskl\") pod \"authorino-operator-75cc8d5cb7-sv9jd\" (UID: \"ab3e808c-304a-4461-8a64-64323dc1042c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:46.247563 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.247531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:46.382459 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.382426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"]
Jul 17 15:17:46.386604 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:17:46.386571    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3e808c_304a_4461_8a64_64323dc1042c.slice/crio-38fbd106df782ab45d05599d09380e69eff68ad451ff96ef573931caf4a78157 WatchSource:0}: Error finding container 38fbd106df782ab45d05599d09380e69eff68ad451ff96ef573931caf4a78157: Status 404 returned error can't find the container with id 38fbd106df782ab45d05599d09380e69eff68ad451ff96ef573931caf4a78157
Jul 17 15:17:46.521286 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:46.521198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd" event={"ID":"ab3e808c-304a-4461-8a64-64323dc1042c","Type":"ContainerStarted","Data":"38fbd106df782ab45d05599d09380e69eff68ad451ff96ef573931caf4a78157"}
Jul 17 15:17:49.540976 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:49.540944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd" event={"ID":"ab3e808c-304a-4461-8a64-64323dc1042c","Type":"ContainerStarted","Data":"580dcd4bbdb4e5bb4a20d101c511409a663e75aa7b706459a976b742f126dcfe"}
Jul 17 15:17:49.541395 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:49.541178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:17:49.559880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:17:49.559495    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd" podStartSLOduration=1.99977347 podStartE2EDuration="4.559473456s" totalImagesPullingTime="2.559699986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:17:49.55864945 +0000 UTC m=+519.833797772" watchObservedRunningTime="2026-07-17 15:17:49.559473456 +0000 UTC m=+519.834621778"
Jul 17 15:18:00.548412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:00.548322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sv9jd"
Jul 17 15:18:34.105362 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.105326    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:34.107436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.107414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:34.109426 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.109401    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2pp5v\""
Jul 17 15:18:34.116762 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.116742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:34.172267 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.172231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f999\" (UniqueName: \"kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999\") pod \"authorino-58cd955675-5lvtf\" (UID: \"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df\") " pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:34.272941 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.272904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f999\" (UniqueName: \"kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999\") pod \"authorino-58cd955675-5lvtf\" (UID: \"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df\") " pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:34.281441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.281407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f999\" (UniqueName: \"kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999\") pod \"authorino-58cd955675-5lvtf\" (UID: \"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df\") " pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:34.426489 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.426401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:34.558918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.558884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:34.562292 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:18:34.562263    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ce7cf5_a3a3_4dec_b52e_282b1509b3df.slice/crio-14b5fa73710be996122face524d5de28775facd759bb35666b63bd9bb4dcfcce WatchSource:0}: Error finding container 14b5fa73710be996122face524d5de28775facd759bb35666b63bd9bb4dcfcce: Status 404 returned error can't find the container with id 14b5fa73710be996122face524d5de28775facd759bb35666b63bd9bb4dcfcce
Jul 17 15:18:34.738565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:34.738534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5lvtf" event={"ID":"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df","Type":"ContainerStarted","Data":"14b5fa73710be996122face524d5de28775facd759bb35666b63bd9bb4dcfcce"}
Jul 17 15:18:37.755404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:37.755368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5lvtf" event={"ID":"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df","Type":"ContainerStarted","Data":"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"}
Jul 17 15:18:37.778400 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:37.778343    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5lvtf" podStartSLOduration=0.786775366 podStartE2EDuration="3.778325012s" totalImagesPullingTime="2.991549646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:18:37.776743372 +0000 UTC m=+568.051891690" watchObservedRunningTime="2026-07-17 15:18:37.778325012 +0000 UTC m=+568.053473332"
Jul 17 15:18:41.340637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.340607    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:41.341095 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.340836    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5lvtf" podUID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" containerName="authorino" containerID="cri-o://9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57" gracePeriod=30
Jul 17 15:18:41.589463 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.589438    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:41.743132 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.743096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f999\" (UniqueName: \"kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999\") pod \"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df\" (UID: \"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df\") "
Jul 17 15:18:41.745583 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.745549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999" (OuterVolumeSpecName: "kube-api-access-9f999") pod "d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" (UID: "d9ce7cf5-a3a3-4dec-b52e-282b1509b3df"). InnerVolumeSpecName "kube-api-access-9f999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:18:41.774726 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.774689    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" containerID="9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57" exitCode=0
Jul 17 15:18:41.774891 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.774732    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5lvtf"
Jul 17 15:18:41.774891 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.774747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5lvtf" event={"ID":"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df","Type":"ContainerDied","Data":"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"}
Jul 17 15:18:41.774891 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.774785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5lvtf" event={"ID":"d9ce7cf5-a3a3-4dec-b52e-282b1509b3df","Type":"ContainerDied","Data":"14b5fa73710be996122face524d5de28775facd759bb35666b63bd9bb4dcfcce"}
Jul 17 15:18:41.774891 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.774806    2566 scope.go:117] "RemoveContainer" containerID="9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"
Jul 17 15:18:41.783515 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.783498    2566 scope.go:117] "RemoveContainer" containerID="9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"
Jul 17 15:18:41.783800 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:18:41.783781    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57\": container with ID starting with 9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57 not found: ID does not exist" containerID="9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"
Jul 17 15:18:41.783842 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.783810    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57"} err="failed to get container status \"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57\": rpc error: code = NotFound desc = could not find container \"9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57\": container with ID starting with 9287265e1b63f19d8405b6a9093a821a98fc72b2b3993d24bbaada176a257f57 not found: ID does not exist"
Jul 17 15:18:41.798537 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.798505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:41.804109 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.804085    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5lvtf"]
Jul 17 15:18:41.844137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:41.844108    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f999\" (UniqueName: \"kubernetes.io/projected/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df-kube-api-access-9f999\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:18:42.240740 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:42.240709    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" path="/var/lib/kubelet/pods/d9ce7cf5-a3a3-4dec-b52e-282b1509b3df/volumes"
Jul 17 15:18:54.874911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.874876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"]
Jul 17 15:18:54.875346 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.875235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" containerName="authorino"
Jul 17 15:18:54.875346 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.875247    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" containerName="authorino"
Jul 17 15:18:54.875346 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.875301    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ce7cf5-a3a3-4dec-b52e-282b1509b3df" containerName="authorino"
Jul 17 15:18:54.878181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.878158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:54.880312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.880291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 15:18:54.880414 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.880294    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 15:18:54.880697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.880675    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gzjc9\""
Jul 17 15:18:54.880855 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.880837    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 15:18:54.887825 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.887800    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"]
Jul 17 15:18:54.967250 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.967213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54lf\" (UniqueName: \"kubernetes.io/projected/53953c56-7fd3-4dd9-9702-a2dac05597fc-kube-api-access-x54lf\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:54.967250 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:54.967253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/53953c56-7fd3-4dd9-9702-a2dac05597fc-data\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.067816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.067782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/53953c56-7fd3-4dd9-9702-a2dac05597fc-data\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.067988 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.067887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54lf\" (UniqueName: \"kubernetes.io/projected/53953c56-7fd3-4dd9-9702-a2dac05597fc-kube-api-access-x54lf\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.068176 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.068158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/53953c56-7fd3-4dd9-9702-a2dac05597fc-data\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.078178 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.078147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54lf\" (UniqueName: \"kubernetes.io/projected/53953c56-7fd3-4dd9-9702-a2dac05597fc-kube-api-access-x54lf\") pod \"postgres-69fb4594cf-5hrr8\" (UID: \"53953c56-7fd3-4dd9-9702-a2dac05597fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.188921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.188828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:18:55.328701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.328678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"]
Jul 17 15:18:55.331179 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:18:55.331151    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53953c56_7fd3_4dd9_9702_a2dac05597fc.slice/crio-dde742980bad55a2f7ba0200a0ef018a12e85437eaf79fcbabac554003ce3a23 WatchSource:0}: Error finding container dde742980bad55a2f7ba0200a0ef018a12e85437eaf79fcbabac554003ce3a23: Status 404 returned error can't find the container with id dde742980bad55a2f7ba0200a0ef018a12e85437eaf79fcbabac554003ce3a23
Jul 17 15:18:55.839326 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:18:55.839290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8" event={"ID":"53953c56-7fd3-4dd9-9702-a2dac05597fc","Type":"ContainerStarted","Data":"dde742980bad55a2f7ba0200a0ef018a12e85437eaf79fcbabac554003ce3a23"}
Jul 17 15:19:00.866846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:00.866809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8" event={"ID":"53953c56-7fd3-4dd9-9702-a2dac05597fc","Type":"ContainerStarted","Data":"5019d43f169579faa014320f72a5e99a04d901d075e29f3c38ea7d90247c3720"}
Jul 17 15:19:00.867276 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:00.866952    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:19:00.884881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:00.884826    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8" podStartSLOduration=1.865161932 podStartE2EDuration="6.884808375s" totalImagesPullingTime="5.019646443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:19:00.883003603 +0000 UTC m=+591.158151923" watchObservedRunningTime="2026-07-17 15:19:00.884808375 +0000 UTC m=+591.159956715"
Jul 17 15:19:06.901579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:06.901547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5hrr8"
Jul 17 15:19:08.058742 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.058694    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:19:08.064872 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.064839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.067281 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.067259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2pp5v\""
Jul 17 15:19:08.067830 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.067810    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 15:19:08.075080 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.075040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:19:08.186451 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.186416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8k6\" (UniqueName: \"kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.186626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.186458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.287785 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.287749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.287982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.287839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8k6\" (UniqueName: \"kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.290359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.290332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.297444 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.297423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8k6\" (UniqueName: \"kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6\") pod \"authorino-855c889d57-b68b6\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") " pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.384608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.384509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:19:08.538627 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.538587    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:19:08.542407 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:19:08.542368    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc97de7_0a5f_4986_a883_dd681a29ce9b.slice/crio-d1c8d9837c0d1742c1e5a0adfcd9a3a6b73000fbfeba4e416803f4f4e54cdda5 WatchSource:0}: Error finding container d1c8d9837c0d1742c1e5a0adfcd9a3a6b73000fbfeba4e416803f4f4e54cdda5: Status 404 returned error can't find the container with id d1c8d9837c0d1742c1e5a0adfcd9a3a6b73000fbfeba4e416803f4f4e54cdda5
Jul 17 15:19:08.901614 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:08.901575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855c889d57-b68b6" event={"ID":"abc97de7-0a5f-4986-a883-dd681a29ce9b","Type":"ContainerStarted","Data":"d1c8d9837c0d1742c1e5a0adfcd9a3a6b73000fbfeba4e416803f4f4e54cdda5"}
Jul 17 15:19:09.907870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:09.907832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855c889d57-b68b6" event={"ID":"abc97de7-0a5f-4986-a883-dd681a29ce9b","Type":"ContainerStarted","Data":"5979c320b786252ce0efc9d5f00f83043954720f78ae21cc6f2657e2ba85f176"}
Jul 17 15:19:09.925997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:09.925944    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-855c889d57-b68b6" podStartSLOduration=1.409773356 podStartE2EDuration="1.925926609s" totalImagesPullingTime="516.153253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:19:09.924832117 +0000 UTC m=+600.199980437" watchObservedRunningTime="2026-07-17 15:19:09.925926609 +0000 UTC m=+600.201074935"
Jul 17 15:19:10.137314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:10.137281    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:19:10.137839 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:19:10.137819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:20:16.406363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.406322    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"]
Jul 17 15:20:16.409716 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.409694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.412330 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.412307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fwpbb\""
Jul 17 15:20:16.412458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.412308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 15:20:16.412458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.412308    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 15:20:16.412458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.412314    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 15:20:16.418676 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.418653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"]
Jul 17 15:20:16.518001 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.517960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8adda1-42c8-4528-aa7b-f136715560e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.518212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.518093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.518212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.518136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.518212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.518178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.518212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.518203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglgg\" (UniqueName: \"kubernetes.io/projected/3b8adda1-42c8-4528-aa7b-f136715560e3-kube-api-access-mglgg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.518384 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.518236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.618973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619014 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglgg\" (UniqueName: \"kubernetes.io/projected/3b8adda1-42c8-4528-aa7b-f136715560e3-kube-api-access-mglgg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8adda1-42c8-4528-aa7b-f136715560e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619314 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619588 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619655 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.619655 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.619610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.621585 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.621562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8adda1-42c8-4528-aa7b-f136715560e3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.621825 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.621808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8adda1-42c8-4528-aa7b-f136715560e3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.630419 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.630395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglgg\" (UniqueName: \"kubernetes.io/projected/3b8adda1-42c8-4528-aa7b-f136715560e3-kube-api-access-mglgg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prbb6\" (UID: \"3b8adda1-42c8-4528-aa7b-f136715560e3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.721515 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.721435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:16.866684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:16.866658    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"]
Jul 17 15:20:16.868541 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:20:16.868514    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8adda1_42c8_4528_aa7b_f136715560e3.slice/crio-d5cef1813723be517d1338654197db8d541f9ee341e4b1427adece47c994a81e WatchSource:0}: Error finding container d5cef1813723be517d1338654197db8d541f9ee341e4b1427adece47c994a81e: Status 404 returned error can't find the container with id d5cef1813723be517d1338654197db8d541f9ee341e4b1427adece47c994a81e
Jul 17 15:20:17.216478 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:17.216311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6" event={"ID":"3b8adda1-42c8-4528-aa7b-f136715560e3","Type":"ContainerStarted","Data":"d5cef1813723be517d1338654197db8d541f9ee341e4b1427adece47c994a81e"}
Jul 17 15:20:18.222335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:18.222286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6" event={"ID":"3b8adda1-42c8-4528-aa7b-f136715560e3","Type":"ContainerStarted","Data":"ba837b2a84f986b809c264abd09f242e1b2aa14f920d36263afd159be96c4a6f"}
Jul 17 15:20:18.243699 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:18.243648    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6" podStartSLOduration=1.063789965 podStartE2EDuration="2.243631207s" totalImagesPullingTime="1.179841242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:20:18.241718308 +0000 UTC m=+668.516866629" watchObservedRunningTime="2026-07-17 15:20:18.243631207 +0000 UTC m=+668.518779584"
Jul 17 15:20:26.722402 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:26.722340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:26.722940 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:26.722453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:26.734833 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:26.734802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:20:27.275351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:20:27.275322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prbb6"
Jul 17 15:21:09.909231 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.909149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"]
Jul 17 15:21:09.912477 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.912455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.914561 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.914538    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 15:21:09.922213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.922188    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"]
Jul 17 15:21:09.995177 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.995343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztx7l\" (UniqueName: \"kubernetes.io/projected/f11308d2-2c24-448d-9412-aaf3aedc2158-kube-api-access-ztx7l\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.995343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.995343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.995343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f11308d2-2c24-448d-9412-aaf3aedc2158-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:09.995343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:09.995341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096093 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096303 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx7l\" (UniqueName: \"kubernetes.io/projected/f11308d2-2c24-448d-9412-aaf3aedc2158-kube-api-access-ztx7l\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096303 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096303 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096303 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f11308d2-2c24-448d-9412-aaf3aedc2158-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096303 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.096779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.096678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.098346 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.098327    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 15:21:10.098693 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.098671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f11308d2-2c24-448d-9412-aaf3aedc2158-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.106231 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.106209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx7l\" (UniqueName: \"kubernetes.io/projected/f11308d2-2c24-448d-9412-aaf3aedc2158-kube-api-access-ztx7l\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.109031 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.109011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f11308d2-2c24-448d-9412-aaf3aedc2158-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd\" (UID: \"f11308d2-2c24-448d-9412-aaf3aedc2158\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.222250 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.222172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:10.365894 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.365866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"]
Jul 17 15:21:10.368291 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:21:10.368265    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf11308d2_2c24_448d_9412_aaf3aedc2158.slice/crio-ac5dbac7afd0919647ff56535e1b562d1d8e5e2644fe4a82712f76eb8677288c WatchSource:0}: Error finding container ac5dbac7afd0919647ff56535e1b562d1d8e5e2644fe4a82712f76eb8677288c: Status 404 returned error can't find the container with id ac5dbac7afd0919647ff56535e1b562d1d8e5e2644fe4a82712f76eb8677288c
Jul 17 15:21:10.369696 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.369677    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:21:10.463432 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:10.463398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd" event={"ID":"f11308d2-2c24-448d-9412-aaf3aedc2158","Type":"ContainerStarted","Data":"ac5dbac7afd0919647ff56535e1b562d1d8e5e2644fe4a82712f76eb8677288c"}
Jul 17 15:21:11.469202 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:11.469167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd" event={"ID":"f11308d2-2c24-448d-9412-aaf3aedc2158","Type":"ContainerStarted","Data":"c115dfbf7f4f20bcb1ae1003a637059cb8c1afe9a28bbab7486e868c03879a73"}
Jul 17 15:21:11.488924 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:11.488870    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd" podStartSLOduration=2.310980615 podStartE2EDuration="2.488853357s" totalImagesPullingTime="177.872742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:21:11.486980739 +0000 UTC m=+721.762129060" watchObservedRunningTime="2026-07-17 15:21:11.488853357 +0000 UTC m=+721.764001678"
Jul 17 15:21:20.222608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:20.222558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:20.222608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:20.222613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:20.241041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:20.241009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:21:20.522952 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:21:20.522921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd"
Jul 17 15:22:15.574686 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.574646    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:22:15.575277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.574875    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-855c889d57-b68b6" podUID="abc97de7-0a5f-4986-a883-dd681a29ce9b" containerName="authorino" containerID="cri-o://5979c320b786252ce0efc9d5f00f83043954720f78ae21cc6f2657e2ba85f176" gracePeriod=30
Jul 17 15:22:15.762378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.762343    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc97de7-0a5f-4986-a883-dd681a29ce9b" containerID="5979c320b786252ce0efc9d5f00f83043954720f78ae21cc6f2657e2ba85f176" exitCode=0
Jul 17 15:22:15.762532 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.762413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855c889d57-b68b6" event={"ID":"abc97de7-0a5f-4986-a883-dd681a29ce9b","Type":"ContainerDied","Data":"5979c320b786252ce0efc9d5f00f83043954720f78ae21cc6f2657e2ba85f176"}
Jul 17 15:22:15.832995 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.832925    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:22:15.907241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.907201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj8k6\" (UniqueName: \"kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6\") pod \"abc97de7-0a5f-4986-a883-dd681a29ce9b\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") "
Jul 17 15:22:15.907415 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.907312    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert\") pod \"abc97de7-0a5f-4986-a883-dd681a29ce9b\" (UID: \"abc97de7-0a5f-4986-a883-dd681a29ce9b\") "
Jul 17 15:22:15.909498 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.909462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6" (OuterVolumeSpecName: "kube-api-access-zj8k6") pod "abc97de7-0a5f-4986-a883-dd681a29ce9b" (UID: "abc97de7-0a5f-4986-a883-dd681a29ce9b"). InnerVolumeSpecName "kube-api-access-zj8k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:22:15.917655 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:15.917616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "abc97de7-0a5f-4986-a883-dd681a29ce9b" (UID: "abc97de7-0a5f-4986-a883-dd681a29ce9b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:22:16.008525 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.008488    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abc97de7-0a5f-4986-a883-dd681a29ce9b-tls-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:22:16.008525 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.008516    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8k6\" (UniqueName: \"kubernetes.io/projected/abc97de7-0a5f-4986-a883-dd681a29ce9b-kube-api-access-zj8k6\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:22:16.768291 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.768259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855c889d57-b68b6" event={"ID":"abc97de7-0a5f-4986-a883-dd681a29ce9b","Type":"ContainerDied","Data":"d1c8d9837c0d1742c1e5a0adfcd9a3a6b73000fbfeba4e416803f4f4e54cdda5"}
Jul 17 15:22:16.768291 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.768293    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855c889d57-b68b6"
Jul 17 15:22:16.768788 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.768311    2566 scope.go:117] "RemoveContainer" containerID="5979c320b786252ce0efc9d5f00f83043954720f78ae21cc6f2657e2ba85f176"
Jul 17 15:22:16.789250 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.789214    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:22:16.802215 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:16.802181    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-855c889d57-b68b6"]
Jul 17 15:22:18.240435 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:18.240401    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc97de7-0a5f-4986-a883-dd681a29ce9b" path="/var/lib/kubelet/pods/abc97de7-0a5f-4986-a883-dd681a29ce9b/volumes"
Jul 17 15:22:48.174325 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.174243    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd6b847d-mst57"]
Jul 17 15:22:48.174816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.174661    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc97de7-0a5f-4986-a883-dd681a29ce9b" containerName="authorino"
Jul 17 15:22:48.174816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.174674    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc97de7-0a5f-4986-a883-dd681a29ce9b" containerName="authorino"
Jul 17 15:22:48.174816 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.174741    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc97de7-0a5f-4986-a883-dd681a29ce9b" containerName="authorino"
Jul 17 15:22:48.177711 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.177695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.180024 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.179995    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 15:22:48.180230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.180217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 15:22:48.180730 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.180718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 15:22:48.181121 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.181102    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 15:22:48.181205 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.181118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 15:22:48.181673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.181658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-z5mff\""
Jul 17 15:22:48.185912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.185891    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 15:22:48.192529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.192507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6b847d-mst57"]
Jul 17 15:22:48.297823 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-service-ca\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-oauth-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-trusted-ca-bundle\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.297983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-oauth-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.298022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.298008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tjh\" (UniqueName: \"kubernetes.io/projected/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-kube-api-access-g2tjh\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.398856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.398816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-oauth-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.398856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.398855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tjh\" (UniqueName: \"kubernetes.io/projected/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-kube-api-access-g2tjh\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399110 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.398917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-service-ca\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399110 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.398957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-oauth-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-trusted-ca-bundle\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399823 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-oauth-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399955 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-trusted-ca-bundle\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.399991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.399976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-service-ca\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.401579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.401559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-oauth-config\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.401710 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.401692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-console-serving-cert\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.408550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.408526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tjh\" (UniqueName: \"kubernetes.io/projected/c1aa13de-0ddb-4d14-af86-cceabc6fd8a3-kube-api-access-g2tjh\") pod \"console-5cd6b847d-mst57\" (UID: \"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3\") " pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.487343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.487248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:48.635670 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.635648    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd6b847d-mst57"]
Jul 17 15:22:48.637758 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:22:48.637731    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1aa13de_0ddb_4d14_af86_cceabc6fd8a3.slice/crio-67101e5457ff0393bcae6cccfdb343460810fa5745a335fbd71c9ff66e7ad287 WatchSource:0}: Error finding container 67101e5457ff0393bcae6cccfdb343460810fa5745a335fbd71c9ff66e7ad287: Status 404 returned error can't find the container with id 67101e5457ff0393bcae6cccfdb343460810fa5745a335fbd71c9ff66e7ad287
Jul 17 15:22:48.913224 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.913192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6b847d-mst57" event={"ID":"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3","Type":"ContainerStarted","Data":"ba2bd2803aeebd1981e25033c3dae0ead19c9bb5f2da05e7bda9d20e1081ea74"}
Jul 17 15:22:48.913404 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.913240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd6b847d-mst57" event={"ID":"c1aa13de-0ddb-4d14-af86-cceabc6fd8a3","Type":"ContainerStarted","Data":"67101e5457ff0393bcae6cccfdb343460810fa5745a335fbd71c9ff66e7ad287"}
Jul 17 15:22:48.936139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:48.936081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd6b847d-mst57" podStartSLOduration=0.9360475 podStartE2EDuration="936.0475ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:22:48.935004978 +0000 UTC m=+819.210153299" watchObservedRunningTime="2026-07-17 15:22:48.9360475 +0000 UTC m=+819.211195821"
Jul 17 15:22:58.487959 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:58.487918    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:58.488433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:58.487983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:58.493011 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:58.492986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:22:58.968930 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:22:58.968854    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd6b847d-mst57"
Jul 17 15:24:10.164403 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:24:10.164322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:24:10.166096 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:24:10.166051    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:29:10.194778 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:29:10.194688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:29:10.196462 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:29:10.196437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:34:10.230775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:34:10.230740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:34:10.233040 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:34:10.233019    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:36:03.860736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.860651    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:36:03.863906 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.863884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.865866 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.865844    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e7c78199-openshift-default-dockercfg-xccs9\""
Jul 17 15:36:03.865997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.865897    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 15:36:03.884982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.884949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:36:03.964028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.963996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvtr\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964417 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:03.964417 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:03.964331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvtr\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065850 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.065983 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.065964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.067775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.067750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.067969 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.067952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.074730 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.074702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.074866 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.074846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvtr\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr\") pod \"e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") " pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.175268 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.175182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:04.321454 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.321428    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:36:04.323734 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:36:04.323706    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615c4e74_3d83_49f8_9569_bcc8fcb345fb.slice/crio-309fbe7d2854b6bab4461cdb3eeae3a8c9729c818a40e81f86efefb036061c2d WatchSource:0}: Error finding container 309fbe7d2854b6bab4461cdb3eeae3a8c9729c818a40e81f86efefb036061c2d: Status 404 returned error can't find the container with id 309fbe7d2854b6bab4461cdb3eeae3a8c9729c818a40e81f86efefb036061c2d
Jul 17 15:36:04.325831 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.325812    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:36:04.556008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:04.555978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerStarted","Data":"309fbe7d2854b6bab4461cdb3eeae3a8c9729c818a40e81f86efefb036061c2d"}
Jul 17 15:36:05.316818 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.316744    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:36:05.320634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.320612    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.323245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.323219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e7c78199-serving-cert\""
Jul 17 15:36:05.323376 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.323321    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:36:05.331220 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.331151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:36:05.380050 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.380015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.380258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.380215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n59f\" (UniqueName: \"kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.380328 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.380282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.481015 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.480967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.481015 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.481012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.481286 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.481161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n59f\" (UniqueName: \"kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.481815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.481788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.484219 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.484194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.492993 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.492954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n59f\" (UniqueName: \"kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f\") pod \"maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.636288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.636196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:05.807837 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:05.807807    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:36:06.799815 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:36:06.799771    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104e83a9_a012_476d_be7e_988dd09c7da6.slice/crio-b9ea53ee37eecd086be74c105000753eccb09c590de283ada0c871d1a703d653 WatchSource:0}: Error finding container b9ea53ee37eecd086be74c105000753eccb09c590de283ada0c871d1a703d653: Status 404 returned error can't find the container with id b9ea53ee37eecd086be74c105000753eccb09c590de283ada0c871d1a703d653
Jul 17 15:36:06.862039 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:06.861996    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:06.862138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:06.862101    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:06.862138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:06.862130    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:07.576079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:07.576018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerStarted","Data":"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"}
Jul 17 15:36:07.577719 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:07.577688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerStarted","Data":"b9ea53ee37eecd086be74c105000753eccb09c590de283ada0c871d1a703d653"}
Jul 17 15:36:07.605587 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:07.605512    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podStartSLOduration=2.069765166 podStartE2EDuration="4.605487275s" totalImagesPullingTime="2.535722109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:36:07.601880775 +0000 UTC m=+1617.877029107" watchObservedRunningTime="2026-07-17 15:36:07.605487275 +0000 UTC m=+1617.880635598"
Jul 17 15:36:08.175776 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:08.175742    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:08.181242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:08.181213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:08.584217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:08.584181    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:08.585421 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:08.585398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:09.591117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:09.591084    2566 generic.go:356] "Generic (PLEG): container finished" podID="104e83a9-a012-476d-be7e-988dd09c7da6" containerID="df90f2b8f27b636de378c5a165e00308119ee9f27e6e5881397b2219610db6d2" exitCode=1
Jul 17 15:36:09.591582 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:09.591146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"df90f2b8f27b636de378c5a165e00308119ee9f27e6e5881397b2219610db6d2"}
Jul 17 15:36:09.591582 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:09.591371    2566 scope.go:117] "RemoveContainer" containerID="df90f2b8f27b636de378c5a165e00308119ee9f27e6e5881397b2219610db6d2"
Jul 17 15:36:10.597595 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:10.597556    2566 generic.go:356] "Generic (PLEG): container finished" podID="104e83a9-a012-476d-be7e-988dd09c7da6" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac" exitCode=1
Jul 17 15:36:10.598085 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:10.597637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"}
Jul 17 15:36:10.598085 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:10.597717    2566 scope.go:117] "RemoveContainer" containerID="df90f2b8f27b636de378c5a165e00308119ee9f27e6e5881397b2219610db6d2"
Jul 17 15:36:10.598085 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:10.597926    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:10.598274 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:10.598185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:10.636814 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:10.636766    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:11.607336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:11.607300    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:11.607844 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:11.607595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:12.614874 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:12.614245    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:12.614874 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:12.614501    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:15.636720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:15.636681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:15.637154 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:15.637139    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:15.637351 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:15.637332    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:18.645893 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.645806    2566 generic.go:356] "Generic (PLEG): container finished" podID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerID="a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62" exitCode=0
Jul 17 15:36:18.645893 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.645881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerDied","Data":"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"}
Jul 17 15:36:18.646356 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.646290    2566 scope.go:117] "RemoveContainer" containerID="a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"
Jul 17 15:36:18.647349 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.647322    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:18.647419 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.647388    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:18.647459 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:18.647420    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:36:19.651956 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:19.651923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerStarted","Data":"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"}
Jul 17 15:36:20.175872 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:20.175829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:20.177097 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:20.177068    2566 patch_prober.go:28] interesting pod/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 17 15:36:20.177216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:20.177121    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 17 15:36:21.175614 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:21.175572    2566 patch_prober.go:28] interesting pod/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 17 15:36:21.176020 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:21.175629    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 17 15:36:22.175963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:22.175920    2566 patch_prober.go:28] interesting pod/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 17 15:36:22.176435 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:22.175985    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 17 15:36:23.176174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:23.176116    2566 patch_prober.go:28] interesting pod/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Jul 17 15:36:23.176575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:23.176200    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 17 15:36:23.584137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:23.584100    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:24.180236 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:24.180206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:24.181108 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:24.181086    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:36:29.235558 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:29.235518    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:29.701375 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:29.701330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerStarted","Data":"277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"}
Jul 17 15:36:29.701598 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:29.701577    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:29.720658 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:29.720589    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podStartSLOduration=21.620357478 podStartE2EDuration="24.720567799s" totalImagesPullingTime="3.100210321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:36:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:36:29.720217333 +0000 UTC m=+1639.995365655" watchObservedRunningTime="2026-07-17 15:36:29.720567799 +0000 UTC m=+1639.995716120"
Jul 17 15:36:30.707772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:30.707740    2566 generic.go:356] "Generic (PLEG): container finished" podID="104e83a9-a012-476d-be7e-988dd09c7da6" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893" exitCode=1
Jul 17 15:36:30.708279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:30.707832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"}
Jul 17 15:36:30.708279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:30.707895    2566 scope.go:117] "RemoveContainer" containerID="67e2af0621e7e3cc8655dfe6f0af032105bdef9c0b48df518fa7952150ff9eac"
Jul 17 15:36:30.708279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:30.708213    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:36:30.708407 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:30.708390    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:31.714420 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:31.714379    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:36:31.714811 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:31.714565    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:35.636661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:35.636619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:36:35.637206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:35.637012    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:36:35.637259 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:35.637224    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:36:49.235788 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:36:49.235750    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:36:49.236310 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:36:49.236004    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:00.237451 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:00.237417    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:37:00.852132 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:00.852034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerStarted","Data":"bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"}
Jul 17 15:37:00.852307 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:00.852266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:37:01.858836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:01.858800    2566 generic.go:356] "Generic (PLEG): container finished" podID="104e83a9-a012-476d-be7e-988dd09c7da6" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092" exitCode=1
Jul 17 15:37:01.859262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:01.858851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"}
Jul 17 15:37:01.859262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:01.858909    2566 scope.go:117] "RemoveContainer" containerID="277210fd9f0e19bd6f3337777d9892b54aa073a3d263313b8995d70a1b972893"
Jul 17 15:37:01.859262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:01.859118    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:01.859384 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:01.859355    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:02.865496 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:02.865468    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:02.865978 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:02.865702    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:05.636773 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:05.636728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:37:05.637238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:05.637189    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:05.637382 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:05.637362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:17.235533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:17.235500    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:17.235988 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:17.235669    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:29.235195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:29.235161    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:29.235597 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:29.235345    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:40.239258 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:40.239187    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:40.239697 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:40.239372    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:52.235840 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:52.235809    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:53.099209 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:53.099126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerStarted","Data":"c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"}
Jul 17 15:37:53.099364 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:53.099334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:37:54.104916 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:54.104884    2566 generic.go:356] "Generic (PLEG): container finished" podID="104e83a9-a012-476d-be7e-988dd09c7da6" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92" exitCode=1
Jul 17 15:37:54.105436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:54.104962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"}
Jul 17 15:37:54.105436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:54.105030    2566 scope.go:117] "RemoveContainer" containerID="bb1e99739412557291e012a2c02cf841fe097136aac1ec0c0258ca188d1c1092"
Jul 17 15:37:54.105436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:54.105320    2566 scope.go:117] "RemoveContainer" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"
Jul 17 15:37:54.105577 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:54.105536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:55.114296 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:55.114263    2566 scope.go:117] "RemoveContainer" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"
Jul 17 15:37:55.114706 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:55.114446    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:37:55.636828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:55.636778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:37:56.118714 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:37:56.118690    2566 scope.go:117] "RemoveContainer" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"
Jul 17 15:37:56.119132 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:37:56.118860    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:38:10.240815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:10.240783    2566 scope.go:117] "RemoveContainer" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"
Jul 17 15:38:10.241664 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:10.241633    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn_odh-ai-gateway-infra(104e83a9-a012-476d-be7e-988dd09c7da6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" podUID="104e83a9-a012-476d-be7e-988dd09c7da6"
Jul 17 15:38:20.748253 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.748209    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:38:20.880970 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.880938    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:38:20.947927 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.947891    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca\") pod \"104e83a9-a012-476d-be7e-988dd09c7da6\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") "
Jul 17 15:38:20.948150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.947946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls\") pod \"104e83a9-a012-476d-be7e-988dd09c7da6\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") "
Jul 17 15:38:20.948150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.948016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n59f\" (UniqueName: \"kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f\") pod \"104e83a9-a012-476d-be7e-988dd09c7da6\" (UID: \"104e83a9-a012-476d-be7e-988dd09c7da6\") "
Jul 17 15:38:20.948401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.948374    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "104e83a9-a012-476d-be7e-988dd09c7da6" (UID: "104e83a9-a012-476d-be7e-988dd09c7da6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:38:20.950427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.950399    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "104e83a9-a012-476d-be7e-988dd09c7da6" (UID: "104e83a9-a012-476d-be7e-988dd09c7da6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:38:20.950501 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:20.950427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f" (OuterVolumeSpecName: "kube-api-access-2n59f") pod "104e83a9-a012-476d-be7e-988dd09c7da6" (UID: "104e83a9-a012-476d-be7e-988dd09c7da6"). InnerVolumeSpecName "kube-api-access-2n59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:38:21.049155 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.049036    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n59f\" (UniqueName: \"kubernetes.io/projected/104e83a9-a012-476d-be7e-988dd09c7da6-kube-api-access-2n59f\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:21.049155 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.049096    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/104e83a9-a012-476d-be7e-988dd09c7da6-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:21.049155 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.049107    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/104e83a9-a012-476d-be7e-988dd09c7da6-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:21.235923 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.235886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn" event={"ID":"104e83a9-a012-476d-be7e-988dd09c7da6","Type":"ContainerDied","Data":"b9ea53ee37eecd086be74c105000753eccb09c590de283ada0c871d1a703d653"}
Jul 17 15:38:21.235923 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.235914    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"
Jul 17 15:38:21.236195 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.235947    2566 scope.go:117] "RemoveContainer" containerID="c4f4274af914bb3faf594393657e2b0dd4db35d438c33e224b6080ad45716f92"
Jul 17 15:38:21.258120 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.258084    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:38:21.262853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:21.262827    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7c78199-6fd457cbc-vrkzn"]
Jul 17 15:38:22.240921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:22.240883    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" path="/var/lib/kubelet/pods/104e83a9-a012-476d-be7e-988dd09c7da6/volumes"
Jul 17 15:38:35.611093 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:35.611036    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:38:35.611679 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:35.611360    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" containerID="cri-o://626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa" gracePeriod=30
Jul 17 15:38:36.210180 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210148    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:38:36.210510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210498    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210512    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210523    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210528    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210540    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210546    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210698 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210603    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210698 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210609    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.210698 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.210615    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:36.214974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.214952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.219493 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.219468    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c8e11152-gw-openshift-default-dockercfg-s29f5\""
Jul 17 15:38:36.229892 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.229866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:38:36.398397 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398397 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2xh\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.398751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.398728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.499894 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.499856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.499915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500327 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500327 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500327 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500476 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2xh\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500476 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500476 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500629 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.500975 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.500714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.502954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.502933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.503183 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.503166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.509324 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.509300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2xh\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.509456 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.509355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token\") pod \"e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") " pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.528623 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.528593    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:36.671556 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.671532    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:38:36.673843 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:38:36.673814    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f283f55_d0fb_4ae5_a410_f394bf404283.slice/crio-43b43a3d8bab20f5214bd5525810dc1c56240d525d90512b48cc69b62b6c7ce9 WatchSource:0}: Error finding container 43b43a3d8bab20f5214bd5525810dc1c56240d525d90512b48cc69b62b6c7ce9: Status 404 returned error can't find the container with id 43b43a3d8bab20f5214bd5525810dc1c56240d525d90512b48cc69b62b6c7ce9
Jul 17 15:38:36.676026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.675995    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:36.676117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.676103    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:36.676167 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:36.676139    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:37.309652 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.309617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerStarted","Data":"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"}
Jul 17 15:38:37.309893 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.309671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerStarted","Data":"43b43a3d8bab20f5214bd5525810dc1c56240d525d90512b48cc69b62b6c7ce9"}
Jul 17 15:38:37.329986 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.329922    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podStartSLOduration=1.32990048 podStartE2EDuration="1.32990048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:38:37.329133041 +0000 UTC m=+1767.604281366" watchObservedRunningTime="2026-07-17 15:38:37.32990048 +0000 UTC m=+1767.605048803"
Jul 17 15:38:37.529588 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.529557    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:37.534940 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.534912    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:37.544876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.544844    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:38:37.545311 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.545293    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:37.545407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.545312    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:37.545461 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.545447    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:38:37.553927 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.553902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.556041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.556010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c8e11152-serving-cert\""
Jul 17 15:38:37.556212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.556074    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:38:37.565426 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.565387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:38:37.612486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.612451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ztl\" (UniqueName: \"kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.612486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.612485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.612735 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.612514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.713358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.713320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ztl\" (UniqueName: \"kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.713358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.713365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.713927 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.713411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.713927 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:37.713601    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-serving-cert: secret "maas-api-e2e-migrate-c8e11152-serving-cert" not found
Jul 17 15:38:37.713927 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:37.713680    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls podName:904d862e-9d38-4681-84e3-bb93486631dd nodeName:}" failed. No retries permitted until 2026-07-17 15:38:38.213656335 +0000 UTC m=+1768.488804638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls") pod "maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" (UID: "904d862e-9d38-4681-84e3-bb93486631dd") : secret "maas-api-e2e-migrate-c8e11152-serving-cert" not found
Jul 17 15:38:37.714163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.714143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:37.722706 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:37.722680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ztl\" (UniqueName: \"kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:38.217112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.217074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:38.219903 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.219870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") pod \"maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:38.314983 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.314953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:38.316090 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.316044    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:38.467632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.467528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:38.607235 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:38.607210    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:38:38.609541 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:38:38.609510    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904d862e_9d38_4681_84e3_bb93486631dd.slice/crio-c79dacc21ad6f7876d9b016ed09cd6b25e3fb143c9369624bbf07cc3696072a9 WatchSource:0}: Error finding container c79dacc21ad6f7876d9b016ed09cd6b25e3fb143c9369624bbf07cc3696072a9: Status 404 returned error can't find the container with id c79dacc21ad6f7876d9b016ed09cd6b25e3fb143c9369624bbf07cc3696072a9
Jul 17 15:38:39.182766 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.182730    2566 patch_prober.go:28] interesting pod/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:38:39.183142 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.182791    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:38:39.321432 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.321397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerStarted","Data":"e5f2bfd3074512d50feb89eec6d63de74efd8138a87c40a1b1bf9f1f9499ce68"}
Jul 17 15:38:39.321605 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.321451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerStarted","Data":"c79dacc21ad6f7876d9b016ed09cd6b25e3fb143c9369624bbf07cc3696072a9"}
Jul 17 15:38:39.321654 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.321615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:39.341911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:39.341858    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podStartSLOduration=1.9669320030000002 podStartE2EDuration="2.341839584s" totalImagesPullingTime="374.907581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:38:39.340043999 +0000 UTC m=+1769.615192320" watchObservedRunningTime="2026-07-17 15:38:39.341839584 +0000 UTC m=+1769.616987904"
Jul 17 15:38:40.327856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.327821    2566 generic.go:356] "Generic (PLEG): container finished" podID="904d862e-9d38-4681-84e3-bb93486631dd" containerID="e5f2bfd3074512d50feb89eec6d63de74efd8138a87c40a1b1bf9f1f9499ce68" exitCode=1
Jul 17 15:38:40.328395 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.327904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerDied","Data":"e5f2bfd3074512d50feb89eec6d63de74efd8138a87c40a1b1bf9f1f9499ce68"}
Jul 17 15:38:40.328395 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.328183    2566 scope.go:117] "RemoveContainer" containerID="e5f2bfd3074512d50feb89eec6d63de74efd8138a87c40a1b1bf9f1f9499ce68"
Jul 17 15:38:40.880442 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.880372    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:38:40.940433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940632 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxvtr\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:38:40.940858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940739    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.940858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data\") pod \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\" (UID: \"615c4e74-3d83-49f8-9569-bcc8fcb345fb\") "
Jul 17 15:38:40.941152 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.940906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:38:40.941152 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941028    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:38:40.941274 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941211    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:40.941274 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941230    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:40.941274 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941231    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:38:40.941274 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941244    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:40.941730 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.941701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:38:40.943685 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.943657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:38:40.943808 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.943661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:38:40.943808 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.943775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:38:40.943986 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:40.943892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr" (OuterVolumeSpecName: "kube-api-access-lxvtr") pod "615c4e74-3d83-49f8-9569-bcc8fcb345fb" (UID: "615c4e74-3d83-49f8-9569-bcc8fcb345fb"). InnerVolumeSpecName "kube-api-access-lxvtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:38:41.041646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041607    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.041646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041640    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.041646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041650    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.041646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041658    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.041927 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041669    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvtr\" (UniqueName: \"kubernetes.io/projected/615c4e74-3d83-49f8-9569-bcc8fcb345fb-kube-api-access-lxvtr\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.041927 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.041678    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615c4e74-3d83-49f8-9569-bcc8fcb345fb-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:38:41.333886 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.333851    2566 generic.go:356] "Generic (PLEG): container finished" podID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerID="626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa" exitCode=0
Jul 17 15:38:41.334374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.333902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"
Jul 17 15:38:41.334374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.333941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerDied","Data":"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"}
Jul 17 15:38:41.334374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.334012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8" event={"ID":"615c4e74-3d83-49f8-9569-bcc8fcb345fb","Type":"ContainerDied","Data":"309fbe7d2854b6bab4461cdb3eeae3a8c9729c818a40e81f86efefb036061c2d"}
Jul 17 15:38:41.334374 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.334046    2566 scope.go:117] "RemoveContainer" containerID="626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"
Jul 17 15:38:41.335989 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.335965    2566 generic.go:356] "Generic (PLEG): container finished" podID="904d862e-9d38-4681-84e3-bb93486631dd" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1" exitCode=1
Jul 17 15:38:41.336147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.336025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerDied","Data":"b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"}
Jul 17 15:38:41.336472 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.336453    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:38:41.336740 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:41.336714    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:38:41.343687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.343668    2566 scope.go:117] "RemoveContainer" containerID="a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"
Jul 17 15:38:41.352432 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.352409    2566 scope.go:117] "RemoveContainer" containerID="626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"
Jul 17 15:38:41.352750 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:41.352730    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa\": container with ID starting with 626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa not found: ID does not exist" containerID="626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"
Jul 17 15:38:41.352799 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.352763    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa"} err="failed to get container status \"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa\": rpc error: code = NotFound desc = could not find container \"626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa\": container with ID starting with 626cdeaf191b038d8fa0b7199e4b5984b2d462dc86a9333e33302347546293fa not found: ID does not exist"
Jul 17 15:38:41.352799 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.352784    2566 scope.go:117] "RemoveContainer" containerID="a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"
Jul 17 15:38:41.353044 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:41.353021    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62\": container with ID starting with a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62 not found: ID does not exist" containerID="a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"
Jul 17 15:38:41.353165 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.353068    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62"} err="failed to get container status \"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62\": rpc error: code = NotFound desc = could not find container \"a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62\": container with ID starting with a05b297c54f3182fa3952c42332b30f6f3022586456bf607f3bc338a1156ba62 not found: ID does not exist"
Jul 17 15:38:41.353165 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.353093    2566 scope.go:117] "RemoveContainer" containerID="e5f2bfd3074512d50feb89eec6d63de74efd8138a87c40a1b1bf9f1f9499ce68"
Jul 17 15:38:41.379828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.379774    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:38:41.380865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:41.380840    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e7c78199-openshift-default-6864558f65-j5xb8"]
Jul 17 15:38:42.242267 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:42.242234    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" path="/var/lib/kubelet/pods/615c4e74-3d83-49f8-9569-bcc8fcb345fb/volumes"
Jul 17 15:38:42.342641 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:42.342616    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:38:42.343018 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:42.342806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:38:44.322561 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:44.322526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:44.322981 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:44.322943    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:38:44.323185 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:44.323165    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:38:48.467853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:48.467804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:38:48.468344 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:48.468329    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:38:48.468576 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:38:48.468554    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:38:50.383950 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.383917    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerID="3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd" exitCode=0
Jul 17 15:38:50.384437 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.383989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerDied","Data":"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"}
Jul 17 15:38:50.384437 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.384422    2566 scope.go:117] "RemoveContainer" containerID="3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"
Jul 17 15:38:50.385502 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.385472    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:50.385618 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.385542    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:50.385618 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:50.385573    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:38:51.390332 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:51.390294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerStarted","Data":"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"}
Jul 17 15:38:51.529383 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:51.529344    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:51.530390 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:51.530357    2566 patch_prober.go:28] interesting pod/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Jul 17 15:38:51.530516 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:51.530411    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Jul 17 15:38:52.529618 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:52.529582    2566 patch_prober.go:28] interesting pod/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Jul 17 15:38:52.530031 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:52.529641    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Jul 17 15:38:53.315078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:53.315010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:53.529230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:53.529194    2566 patch_prober.go:28] interesting pod/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Jul 17 15:38:53.529409 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:53.529251    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Jul 17 15:38:54.529212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:54.529176    2566 patch_prober.go:28] interesting pod/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Jul 17 15:38:54.529606 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:54.529232    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Jul 17 15:38:55.533604 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:55.533573    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:55.534768 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:55.534750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:38:59.235088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:38:59.235019    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:39:00.441467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:00.441348    2566 generic.go:356] "Generic (PLEG): container finished" podID="904d862e-9d38-4681-84e3-bb93486631dd" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6" exitCode=1
Jul 17 15:39:00.441467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:00.441422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerDied","Data":"37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"}
Jul 17 15:39:00.441895 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:00.441487    2566 scope.go:117] "RemoveContainer" containerID="b0a30896faf148b3345bf9c367d7ecd04558b886994ba314cecb5396c79145e1"
Jul 17 15:39:00.441895 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:00.441875    2566 scope.go:117] "RemoveContainer" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"
Jul 17 15:39:00.442107 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:00.442088    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:04.322122 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:04.322085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:04.322566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:04.322473    2566 scope.go:117] "RemoveContainer" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"
Jul 17 15:39:04.322702 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:04.322682    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:08.468327 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:08.468290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:08.468764 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:08.468721    2566 scope.go:117] "RemoveContainer" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"
Jul 17 15:39:08.468920 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:08.468898    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:10.260983 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:10.260957    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:39:10.264519 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:10.264497    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:39:24.235486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:24.235440    2566 scope.go:117] "RemoveContainer" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"
Jul 17 15:39:25.563163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:25.563134    2566 generic.go:356] "Generic (PLEG): container finished" podID="904d862e-9d38-4681-84e3-bb93486631dd" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7" exitCode=1
Jul 17 15:39:25.563572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:25.563202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerDied","Data":"1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"}
Jul 17 15:39:25.563572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:25.563260    2566 scope.go:117] "RemoveContainer" containerID="37344f5061fac7d09189465c158c28108bd28cd076dce5ef1891573b858b91f6"
Jul 17 15:39:25.563661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:25.563627    2566 scope.go:117] "RemoveContainer" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"
Jul 17 15:39:25.563863 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:25.563840    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:28.468734 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:28.468697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:28.469218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:28.469090    2566 scope.go:117] "RemoveContainer" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"
Jul 17 15:39:28.469306 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:28.469285    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:29.321942 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:29.321895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:29.322308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:29.322294    2566 scope.go:117] "RemoveContainer" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"
Jul 17 15:39:29.322513 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:29.322495    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:42.146954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:42.146911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:39:42.147379 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:42.147179    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy" containerID="cri-o://8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715" gracePeriod=30
Jul 17 15:39:44.235522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:44.235478    2566 scope.go:117] "RemoveContainer" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"
Jul 17 15:39:44.236175 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:44.235691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k_odh-ai-gateway-infra(904d862e-9d38-4681-84e3-bb93486631dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" podUID="904d862e-9d38-4681-84e3-bb93486631dd"
Jul 17 15:39:47.404720 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.404695    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:39:47.446856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.446856 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447092 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps2xh\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447092 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447092 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447092 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.446958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:39:47.447312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447199    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket\") pod \"2f283f55-d0fb-4ae5-a410-f394bf404283\" (UID: \"2f283f55-d0fb-4ae5-a410-f394bf404283\") "
Jul 17 15:39:47.447625 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447602    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.447774 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:39:47.447858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.447529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:39:47.448050 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.448026    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:39:47.448273 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.448248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:39:47.449918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.449867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:39:47.449918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.449884    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh" (OuterVolumeSpecName: "kube-api-access-ps2xh") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "kube-api-access-ps2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:39:47.450263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.450105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:39:47.450496 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.450474    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f283f55-d0fb-4ae5-a410-f394bf404283" (UID: "2f283f55-d0fb-4ae5-a410-f394bf404283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:39:47.548296 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548258    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548296 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548293    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548306    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2xh\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-kube-api-access-ps2xh\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548323    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548335    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f283f55-d0fb-4ae5-a410-f394bf404283-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548345    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548356    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.548522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.548367    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f283f55-d0fb-4ae5-a410-f394bf404283-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:47.674040 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.673999    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerID="8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715" exitCode=0
Jul 17 15:39:47.674040 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.674070    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"
Jul 17 15:39:47.674284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.674111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerDied","Data":"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"}
Jul 17 15:39:47.674284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.674219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6" event={"ID":"2f283f55-d0fb-4ae5-a410-f394bf404283","Type":"ContainerDied","Data":"43b43a3d8bab20f5214bd5525810dc1c56240d525d90512b48cc69b62b6c7ce9"}
Jul 17 15:39:47.674284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.674252    2566 scope.go:117] "RemoveContainer" containerID="8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"
Jul 17 15:39:47.684930 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.684908    2566 scope.go:117] "RemoveContainer" containerID="3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"
Jul 17 15:39:47.693726 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.693705    2566 scope.go:117] "RemoveContainer" containerID="8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"
Jul 17 15:39:47.694099 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:47.694074    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715\": container with ID starting with 8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715 not found: ID does not exist" containerID="8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"
Jul 17 15:39:47.694200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.694110    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715"} err="failed to get container status \"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715\": rpc error: code = NotFound desc = could not find container \"8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715\": container with ID starting with 8af690a998304d319500c4e0cfc3bf59ed46eff9857f630062bc9caa0123b715 not found: ID does not exist"
Jul 17 15:39:47.694200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.694132    2566 scope.go:117] "RemoveContainer" containerID="3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"
Jul 17 15:39:47.694401 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:47.694380    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd\": container with ID starting with 3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd not found: ID does not exist" containerID="3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"
Jul 17 15:39:47.694438 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.694414    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd"} err="failed to get container status \"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd\": rpc error: code = NotFound desc = could not find container \"3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd\": container with ID starting with 3673c064d1d91d38073e69b7780c7f07acc43ab939ad6d6d032f57e5b37697cd not found: ID does not exist"
Jul 17 15:39:47.698389 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.698332    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:39:47.704041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:47.704016    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c8e11152-gw-openshift-default-75cd6b8bd7-ms7m6"]
Jul 17 15:39:48.248165 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.248131    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" path="/var/lib/kubelet/pods/2f283f55-d0fb-4ae5-a410-f394bf404283/volumes"
Jul 17 15:39:48.486884 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.486844    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:39:48.658164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.658140    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:48.682496 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.682468    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"
Jul 17 15:39:48.682671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.682463    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k" event={"ID":"904d862e-9d38-4681-84e3-bb93486631dd","Type":"ContainerDied","Data":"c79dacc21ad6f7876d9b016ed09cd6b25e3fb143c9369624bbf07cc3696072a9"}
Jul 17 15:39:48.682671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.682609    2566 scope.go:117] "RemoveContainer" containerID="1cfb9d7924e70f0215d54d3652356b2bbb60ca6c4ce64411f865020bb0712bf7"
Jul 17 15:39:48.759652 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.759606    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9ztl\" (UniqueName: \"kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl\") pod \"904d862e-9d38-4681-84e3-bb93486631dd\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") "
Jul 17 15:39:48.759804 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.759683    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") pod \"904d862e-9d38-4681-84e3-bb93486631dd\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") "
Jul 17 15:39:48.759804 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.759706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca\") pod \"904d862e-9d38-4681-84e3-bb93486631dd\" (UID: \"904d862e-9d38-4681-84e3-bb93486631dd\") "
Jul 17 15:39:48.760139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.760105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "904d862e-9d38-4681-84e3-bb93486631dd" (UID: "904d862e-9d38-4681-84e3-bb93486631dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:39:48.761906 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.761882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl" (OuterVolumeSpecName: "kube-api-access-n9ztl") pod "904d862e-9d38-4681-84e3-bb93486631dd" (UID: "904d862e-9d38-4681-84e3-bb93486631dd"). InnerVolumeSpecName "kube-api-access-n9ztl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:39:48.761906 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.761891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "904d862e-9d38-4681-84e3-bb93486631dd" (UID: "904d862e-9d38-4681-84e3-bb93486631dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:39:48.861021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.860934    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9ztl\" (UniqueName: \"kubernetes.io/projected/904d862e-9d38-4681-84e3-bb93486631dd-kube-api-access-n9ztl\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:48.861021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.860965    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/904d862e-9d38-4681-84e3-bb93486631dd-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:48.861021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:48.860975    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/904d862e-9d38-4681-84e3-bb93486631dd-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:39:49.004845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:49.004811    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:39:49.009752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:49.009715    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c8e11152-5bfddd847-tll6k"]
Jul 17 15:39:50.241076 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:50.241020    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904d862e-9d38-4681-84e3-bb93486631dd" path="/var/lib/kubelet/pods/904d862e-9d38-4681-84e3-bb93486631dd/volumes"
Jul 17 15:39:55.766682 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.766646    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:39:55.767452 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767412    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767628 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767612    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767706 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767676    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767706 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767689    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767703    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767731    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767746    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767755    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767772    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767781    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:55.767821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767818    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767836    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767860    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767869    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767968    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767983    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.767991    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.768002    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.768011    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e83a9-a012-476d-be7e-988dd09c7da6" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.768020    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:55.768226 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.768030    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f283f55-d0fb-4ae5-a410-f394bf404283" containerName="istio-proxy"
Jul 17 15:39:55.772852 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.772824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.775124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.775042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 15:39:55.775124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.775042    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-969af631-openshift-default-dockercfg-mz9wg\""
Jul 17 15:39:55.784079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.784028    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:39:55.931531 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931721 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931721 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931721 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92x5\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931721 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931721 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931992 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931992 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:55.931992 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:55.931922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92x5\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.032789 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.032804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.033044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033139 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.033044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.033174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.033204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.033505 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.033485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.035341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.035315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.035456 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.035345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.041661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.041638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.041775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.041728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92x5\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5\") pod \"e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") " pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.084869 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.084828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:56.229834 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.229807    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:39:56.232716 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:39:56.232684    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e6c36e_bb10_493a_8e2a_bb4bc1dc7385.slice/crio-dc55b15ffbee876dd514c0b60fc3b95c08bfa7515a4c904db8e9cec1834a4f4e WatchSource:0}: Error finding container dc55b15ffbee876dd514c0b60fc3b95c08bfa7515a4c904db8e9cec1834a4f4e: Status 404 returned error can't find the container with id dc55b15ffbee876dd514c0b60fc3b95c08bfa7515a4c904db8e9cec1834a4f4e
Jul 17 15:39:56.235032 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.234996    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:39:56.235162 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.235088    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:39:56.235162 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.235136    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:39:56.721450 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.721352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerStarted","Data":"8e4ec350014419b5fd2aa6ea17c48610b36a61278b63409bf7cb94bf6ccbd498"}
Jul 17 15:39:56.721450 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.721421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerStarted","Data":"dc55b15ffbee876dd514c0b60fc3b95c08bfa7515a4c904db8e9cec1834a4f4e"}
Jul 17 15:39:56.744443 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:56.744386    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podStartSLOduration=1.7443647310000001 podStartE2EDuration="1.744364731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:39:56.743648945 +0000 UTC m=+1847.018797267" watchObservedRunningTime="2026-07-17 15:39:56.744364731 +0000 UTC m=+1847.019513053"
Jul 17 15:39:57.037262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.037228    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:39:57.037651 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.037597    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:57.037651 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.037609    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:57.037731 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.037680    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="904d862e-9d38-4681-84e3-bb93486631dd" containerName="maas-api"
Jul 17 15:39:57.037731 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.037691    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="615c4e74-3d83-49f8-9569-bcc8fcb345fb" containerName="istio-proxy"
Jul 17 15:39:57.040712 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.040694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.044391 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.044369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-969af631-serving-cert\""
Jul 17 15:39:57.045660 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.045640    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:39:57.073182 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.073150    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:39:57.085961 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.085929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:57.091454 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.091427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:57.144086 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.144018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.144257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.144106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69thj\" (UniqueName: \"kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.144257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.144174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.244744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.244698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.244912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.244778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.244912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.244806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69thj\" (UniqueName: \"kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.245029 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:57.244932    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-969af631-serving-cert: secret "maas-api-e2e-ait-969af631-serving-cert" not found
Jul 17 15:39:57.245029 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:57.245019    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls podName:2670701f-64e1-4380-9a59-3bf7f310834c nodeName:}" failed. No retries permitted until 2026-07-17 15:39:57.744992927 +0000 UTC m=+1848.020141226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls") pod "maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" (UID: "2670701f-64e1-4380-9a59-3bf7f310834c") : secret "maas-api-e2e-ait-969af631-serving-cert" not found
Jul 17 15:39:57.245469 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.245453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.255041 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.255016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69thj\" (UniqueName: \"kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.726752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.726722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:57.727710 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.727679    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:39:57.750121 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:57.750053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:57.750364 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:57.750208    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-969af631-serving-cert: secret "maas-api-e2e-ait-969af631-serving-cert" not found
Jul 17 15:39:57.750364 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:39:57.750286    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls podName:2670701f-64e1-4380-9a59-3bf7f310834c nodeName:}" failed. No retries permitted until 2026-07-17 15:39:58.750265071 +0000 UTC m=+1849.025413373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls") pod "maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" (UID: "2670701f-64e1-4380-9a59-3bf7f310834c") : secret "maas-api-e2e-ait-969af631-serving-cert" not found
Jul 17 15:39:58.758793 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:58.758743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:58.761343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:58.761323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"maas-api-e2e-ait-969af631-7796c96bfc-5cwh9\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:58.852974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:58.852930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:39:59.000292 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:59.000213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:39:59.003109 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:39:59.003086    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670701f_64e1_4380_9a59_3bf7f310834c.slice/crio-e0656f9a224094829ac416f5dbc130497e7f03fe9cf5c6bdc19f45d20c581709 WatchSource:0}: Error finding container e0656f9a224094829ac416f5dbc130497e7f03fe9cf5c6bdc19f45d20c581709: Status 404 returned error can't find the container with id e0656f9a224094829ac416f5dbc130497e7f03fe9cf5c6bdc19f45d20c581709
Jul 17 15:39:59.736248 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:59.736213    2566 generic.go:356] "Generic (PLEG): container finished" podID="2670701f-64e1-4380-9a59-3bf7f310834c" containerID="af57cea8124ea8f5d28a88e2527f534cbb4e0083ee1ec682989d6b22df4f7b48" exitCode=1
Jul 17 15:39:59.736398 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:59.736309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerDied","Data":"af57cea8124ea8f5d28a88e2527f534cbb4e0083ee1ec682989d6b22df4f7b48"}
Jul 17 15:39:59.736398 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:59.736388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerStarted","Data":"e0656f9a224094829ac416f5dbc130497e7f03fe9cf5c6bdc19f45d20c581709"}
Jul 17 15:39:59.736506 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:39:59.736489    2566 scope.go:117] "RemoveContainer" containerID="af57cea8124ea8f5d28a88e2527f534cbb4e0083ee1ec682989d6b22df4f7b48"
Jul 17 15:40:00.743581 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:00.743537    2566 generic.go:356] "Generic (PLEG): container finished" podID="2670701f-64e1-4380-9a59-3bf7f310834c" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c" exitCode=1
Jul 17 15:40:00.744034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:00.743647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerDied","Data":"ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"}
Jul 17 15:40:00.744034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:00.743748    2566 scope.go:117] "RemoveContainer" containerID="af57cea8124ea8f5d28a88e2527f534cbb4e0083ee1ec682989d6b22df4f7b48"
Jul 17 15:40:00.744034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:00.743943    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:00.744223 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:00.744204    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:01.750440 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:01.750406    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:01.750817 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:01.750639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:03.853555 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:03.853504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:03.853969 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:03.853910    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:03.854133 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:03.854113    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:08.787184 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.787150    2566 generic.go:356] "Generic (PLEG): container finished" podID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerID="8e4ec350014419b5fd2aa6ea17c48610b36a61278b63409bf7cb94bf6ccbd498" exitCode=0
Jul 17 15:40:08.787624 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.787226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerDied","Data":"8e4ec350014419b5fd2aa6ea17c48610b36a61278b63409bf7cb94bf6ccbd498"}
Jul 17 15:40:08.787671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.787638    2566 scope.go:117] "RemoveContainer" containerID="8e4ec350014419b5fd2aa6ea17c48610b36a61278b63409bf7cb94bf6ccbd498"
Jul 17 15:40:08.788644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.788617    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:40:08.788697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.788687    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:40:08.788750 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.788738    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:40:08.853329 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.853299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:08.853687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:08.853674    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:08.853879 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:08.853861    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:09.795930 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:09.795892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerStarted","Data":"b029c73cdf196a06531c00c2ad2f9c2bc83d153f43e8bf584dcce5f4b9dcadb6"}
Jul 17 15:40:10.085576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:10.085475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:40:10.086709 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:10.086680    2566 patch_prober.go:28] interesting pod/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 17 15:40:10.086832 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:10.086729    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 17 15:40:11.085673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:11.085633    2566 patch_prober.go:28] interesting pod/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 17 15:40:11.086097 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:11.085691    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 17 15:40:12.086197 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:12.086157    2566 patch_prober.go:28] interesting pod/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 17 15:40:12.086595 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:12.086217    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 17 15:40:12.726697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:12.726659    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:40:13.085460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:13.085424    2566 patch_prober.go:28] interesting pod/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 17 15:40:13.085651 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:13.085481    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 17 15:40:14.090355 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:14.090324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:40:14.091481 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:14.091450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:40:21.235501 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:21.235465    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:22.867123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:22.867087    2566 generic.go:356] "Generic (PLEG): container finished" podID="2670701f-64e1-4380-9a59-3bf7f310834c" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489" exitCode=1
Jul 17 15:40:22.867621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:22.867135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerDied","Data":"a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"}
Jul 17 15:40:22.867621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:22.867182    2566 scope.go:117] "RemoveContainer" containerID="ddd86ba0754ec19e87f87218152049999e572e2fb972b116f697ef1d206c170c"
Jul 17 15:40:22.867621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:22.867571    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:22.867847 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:22.867823    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:23.853554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:23.853512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:23.873385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:23.873360    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:23.873848 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:23.873534    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:28.853864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:28.853826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:28.854361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:28.854256    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:28.854455 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:28.854435    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:40.238630 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:40.238551    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:40.239020 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:40.238761    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:51.235858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:51.235818    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:52.010600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:52.010567    2566 generic.go:356] "Generic (PLEG): container finished" podID="2670701f-64e1-4380-9a59-3bf7f310834c" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429" exitCode=1
Jul 17 15:40:52.010782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:52.010619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerDied","Data":"b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"}
Jul 17 15:40:52.010782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:52.010667    2566 scope.go:117] "RemoveContainer" containerID="a59d9abad8bff854665f8abd749d5fc6400a1fb492dc7dcd4e2600525fb90489"
Jul 17 15:40:52.011078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:52.011046    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:40:52.011287 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:52.011264    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:53.853353 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:53.853317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:53.853813 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:53.853703    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:40:53.853910 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:53.853889    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:40:58.853979 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:58.853936    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:40:58.854419 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:40:58.854372    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:40:58.854563 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:40:58.854542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:41:11.235881 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:41:11.235840    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:41:11.236503 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:41:11.236162    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:41:25.236200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:41:25.236163    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:41:25.236632 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:41:25.236331    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:41:40.238624 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:41:40.238593    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:41:40.239136 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:41:40.238992    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:41:50.645989 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:41:50.645939    2566 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/maas-api@sha256:a201ba1f84c6b2671301b1404c58b15423c9811f0865ed8ac8d4f8861ba3d2cf: reading manifest sha256:716fef4271f4d59a604124a7ca7f2c02512db48b11b68b4e6c31db4fab2f32ce in quay.io/opendatahub/maas-api: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/opendatahub/maas-api@sha256:a201ba1f84c6b2671301b1404c58b15423c9811f0865ed8ac8d4f8861ba3d2cf"
Jul 17 15:41:50.646506 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:41:50.646081    2566 kuberuntime_manager.go:1459] "Unhandled Error" err="container maas-api start failed in pod maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/maas-api@sha256:a201ba1f84c6b2671301b1404c58b15423c9811f0865ed8ac8d4f8861ba3d2cf: reading manifest sha256:716fef4271f4d59a604124a7ca7f2c02512db48b11b68b4e6c31db4fab2f32ce in quay.io/opendatahub/maas-api: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 17 15:41:50.646506 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:41:50.646119    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/maas-api@sha256:a201ba1f84c6b2671301b1404c58b15423c9811f0865ed8ac8d4f8861ba3d2cf: reading manifest sha256:716fef4271f4d59a604124a7ca7f2c02512db48b11b68b4e6c31db4fab2f32ce in quay.io/opendatahub/maas-api: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:42:01.235843 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:01.235752    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:42:01.236369 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:01.236007    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-969af631-7796c96bfc-5cwh9_odh-ai-gateway-infra(2670701f-64e1-4380-9a59-3bf7f310834c)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" podUID="2670701f-64e1-4380-9a59-3bf7f310834c"
Jul 17 15:42:12.737083 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.737036    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:42:12.876803 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.876774    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:42:12.945684 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.945651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca\") pod \"2670701f-64e1-4380-9a59-3bf7f310834c\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") "
Jul 17 15:42:12.945840 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.945699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69thj\" (UniqueName: \"kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj\") pod \"2670701f-64e1-4380-9a59-3bf7f310834c\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") "
Jul 17 15:42:12.945840 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.945768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") pod \"2670701f-64e1-4380-9a59-3bf7f310834c\" (UID: \"2670701f-64e1-4380-9a59-3bf7f310834c\") "
Jul 17 15:42:12.945992 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.945967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2670701f-64e1-4380-9a59-3bf7f310834c" (UID: "2670701f-64e1-4380-9a59-3bf7f310834c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:42:12.947920 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.947894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj" (OuterVolumeSpecName: "kube-api-access-69thj") pod "2670701f-64e1-4380-9a59-3bf7f310834c" (UID: "2670701f-64e1-4380-9a59-3bf7f310834c"). InnerVolumeSpecName "kube-api-access-69thj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:42:12.948093 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:12.948072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2670701f-64e1-4380-9a59-3bf7f310834c" (UID: "2670701f-64e1-4380-9a59-3bf7f310834c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:42:13.046526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.046481    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670701f-64e1-4380-9a59-3bf7f310834c-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:13.046526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.046513    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69thj\" (UniqueName: \"kubernetes.io/projected/2670701f-64e1-4380-9a59-3bf7f310834c-kube-api-access-69thj\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:13.046526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.046524    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2670701f-64e1-4380-9a59-3bf7f310834c-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:13.384186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.384094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9" event={"ID":"2670701f-64e1-4380-9a59-3bf7f310834c","Type":"ContainerDied","Data":"e0656f9a224094829ac416f5dbc130497e7f03fe9cf5c6bdc19f45d20c581709"}
Jul 17 15:42:13.384186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.384154    2566 scope.go:117] "RemoveContainer" containerID="b29496e344edbf77226da2ec49c24d4c1e9cd94a79c4be37be4a7f765bcb5429"
Jul 17 15:42:13.384420 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.384107    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"
Jul 17 15:42:13.406341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.406304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:42:13.411180 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:13.411150    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-969af631-7796c96bfc-5cwh9"]
Jul 17 15:42:14.240821 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:14.240789    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" path="/var/lib/kubelet/pods/2670701f-64e1-4380-9a59-3bf7f310834c/volumes"
Jul 17 15:42:28.321277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.321237    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:42:28.321835 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.321437    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" containerID="cri-o://b029c73cdf196a06531c00c2ad2f9c2bc83d153f43e8bf584dcce5f4b9dcadb6" gracePeriod=30
Jul 17 15:42:28.606150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606029    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:42:28.606614 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606592    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606618    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606635    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606644    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606657    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606666    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606774    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606785    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.606862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.606794    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:28.611867 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.611844    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.614338 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.614303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7726776b-openshift-default-dockercfg-wh5vg\""
Jul 17 15:42:28.621778 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.621753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:42:28.705183 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dz9\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.705602 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.705509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dz9\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.806707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.807072 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.807072 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.807072 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.806987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.807297 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.807279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.807387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.807370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.809050 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.809034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.809302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.809284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.821242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.821206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dz9\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.821242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.821237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token\") pod \"e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") " pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:28.924120 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:28.924013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:29.067150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.067112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:42:29.071017 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:42:29.070990    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75382c25_91aa_4815_9451_e3b84e33e9dd.slice/crio-41f118b9459106f24b724396b8b9b401de6a5f599a2a737e5ad31a67db2a503f WatchSource:0}: Error finding container 41f118b9459106f24b724396b8b9b401de6a5f599a2a737e5ad31a67db2a503f: Status 404 returned error can't find the container with id 41f118b9459106f24b724396b8b9b401de6a5f599a2a737e5ad31a67db2a503f
Jul 17 15:42:29.073176 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.073142    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:29.073288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.073224    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:29.073288 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.073270    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:29.093053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.093023    2566 patch_prober.go:28] interesting pod/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:42:29.093194 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.093103    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:42:29.464652 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.464615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerStarted","Data":"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"}
Jul 17 15:42:29.465072 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.464669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerStarted","Data":"41f118b9459106f24b724396b8b9b401de6a5f599a2a737e5ad31a67db2a503f"}
Jul 17 15:42:29.485234 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.485159    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podStartSLOduration=1.485138053 podStartE2EDuration="1.485138053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:42:29.484877821 +0000 UTC m=+1999.760026168" watchObservedRunningTime="2026-07-17 15:42:29.485138053 +0000 UTC m=+1999.760286407"
Jul 17 15:42:29.837708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.837670    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:42:29.838175 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.838155    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:29.838294 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.838177    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:29.838361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.838293    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2670701f-64e1-4380-9a59-3bf7f310834c" containerName="maas-api"
Jul 17 15:42:29.841385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.841363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:29.843218 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.843201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7726776b-serving-cert\""
Jul 17 15:42:29.843541 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.843528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:42:29.850335 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.850307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:42:29.918831 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.918797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcdm\" (UniqueName: \"kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:29.919018 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.918843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:29.919018 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.918951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:29.924642 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:29.924610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:30.019935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.019893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.020169 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.019995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcdm\" (UniqueName: \"kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.020169 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.020036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.020262 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:30.020240    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-serving-cert: secret "maas-api-e2e-derive-7726776b-serving-cert" not found
Jul 17 15:42:30.020343 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:30.020330    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls podName:f8a49b21-9ac2-4984-9bff-bd5148d9b60a nodeName:}" failed. No retries permitted until 2026-07-17 15:42:30.520307146 +0000 UTC m=+2000.795455450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls") pod "maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" (UID: "f8a49b21-9ac2-4984-9bff-bd5148d9b60a") : secret "maas-api-e2e-derive-7726776b-serving-cert" not found
Jul 17 15:42:30.020683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.020666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.029850 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.029814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcdm\" (UniqueName: \"kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.525835 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.525790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.528381 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.528356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") pod \"maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.753144 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.753108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:30.897357 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.897326    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:42:30.899958 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:42:30.899929    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a49b21_9ac2_4984_9bff_bd5148d9b60a.slice/crio-7ed946c16d2d7b1ff041eaf5d22a6761edd375051b289014ba45c02d508a9cde WatchSource:0}: Error finding container 7ed946c16d2d7b1ff041eaf5d22a6761edd375051b289014ba45c02d508a9cde: Status 404 returned error can't find the container with id 7ed946c16d2d7b1ff041eaf5d22a6761edd375051b289014ba45c02d508a9cde
Jul 17 15:42:30.929794 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:30.929759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:31.475328 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.475296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerStarted","Data":"51ee8a3675d3c260de5bc25dc558016012c831251ecd0d46ad3f9202629bea41"}
Jul 17 15:42:31.475531 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.475351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerStarted","Data":"7ed946c16d2d7b1ff041eaf5d22a6761edd375051b289014ba45c02d508a9cde"}
Jul 17 15:42:31.475531 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.475383    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:31.475635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.475616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:31.476680 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.476657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:31.492360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:31.492302    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podStartSLOduration=2.139252019 podStartE2EDuration="2.492281398s" totalImagesPullingTime="353.029379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:42:31.491288916 +0000 UTC m=+2001.766437266" watchObservedRunningTime="2026-07-17 15:42:31.492281398 +0000 UTC m=+2001.767429723"
Jul 17 15:42:32.481029 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:32.480994    2566 generic.go:356] "Generic (PLEG): container finished" podID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerID="51ee8a3675d3c260de5bc25dc558016012c831251ecd0d46ad3f9202629bea41" exitCode=1
Jul 17 15:42:32.481445 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:32.481079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"51ee8a3675d3c260de5bc25dc558016012c831251ecd0d46ad3f9202629bea41"}
Jul 17 15:42:32.481445 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:32.481337    2566 scope.go:117] "RemoveContainer" containerID="51ee8a3675d3c260de5bc25dc558016012c831251ecd0d46ad3f9202629bea41"
Jul 17 15:42:33.519222 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.519180    2566 generic.go:356] "Generic (PLEG): container finished" podID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb" exitCode=1
Jul 17 15:42:33.524745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.519802    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"}
Jul 17 15:42:33.524745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.523495    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:33.524745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.524370    2566 scope.go:117] "RemoveContainer" containerID="51ee8a3675d3c260de5bc25dc558016012c831251ecd0d46ad3f9202629bea41"
Jul 17 15:42:33.524745 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:33.524447    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:42:33.526700 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.526666    2566 generic.go:356] "Generic (PLEG): container finished" podID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerID="b029c73cdf196a06531c00c2ad2f9c2bc83d153f43e8bf584dcce5f4b9dcadb6" exitCode=0
Jul 17 15:42:33.526937 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.526904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerDied","Data":"b029c73cdf196a06531c00c2ad2f9c2bc83d153f43e8bf584dcce5f4b9dcadb6"}
Jul 17 15:42:33.539499 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.539476    2566 scope.go:117] "RemoveContainer" containerID="8e4ec350014419b5fd2aa6ea17c48610b36a61278b63409bf7cb94bf6ccbd498"
Jul 17 15:42:33.622532 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.622504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:42:33.756576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756580    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j92x5\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.756772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756741    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.757030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.757030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.756818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket\") pod \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\" (UID: \"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385\") "
Jul 17 15:42:33.757176 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.757027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:42:33.757230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.757194    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:42:33.757230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.757219    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.757336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.757317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:42:33.757399 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.757373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:42:33.758078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.758031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:42:33.759123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.759095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:42:33.759233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.759117    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:42:33.759233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.759148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5" (OuterVolumeSpecName: "kube-api-access-j92x5") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "kube-api-access-j92x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:42:33.759233 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.759161    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" (UID: "a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:42:33.858026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.857984    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858017    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858026    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858035    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858043    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858051    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858082    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92x5\" (UniqueName: \"kubernetes.io/projected/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-kube-api-access-j92x5\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:33.858363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:33.858091    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:42:34.533558 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.533471    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:34.533965 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:34.533697    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:42:34.534537 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.534523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"
Jul 17 15:42:34.534609 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.534519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl" event={"ID":"a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385","Type":"ContainerDied","Data":"dc55b15ffbee876dd514c0b60fc3b95c08bfa7515a4c904db8e9cec1834a4f4e"}
Jul 17 15:42:34.534647 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.534640    2566 scope.go:117] "RemoveContainer" containerID="b029c73cdf196a06531c00c2ad2f9c2bc83d153f43e8bf584dcce5f4b9dcadb6"
Jul 17 15:42:34.573036 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.573001    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:42:34.577678 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:34.577653    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-969af631-openshift-default-68b6f8c885-mpxbl"]
Jul 17 15:42:36.241761 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:36.241726    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" path="/var/lib/kubelet/pods/a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385/volumes"
Jul 17 15:42:36.475799 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:36.475761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:36.476210 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:36.476195    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:36.476391 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:36.476372    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:42:40.566467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.566381    2566 generic.go:356] "Generic (PLEG): container finished" podID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerID="5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154" exitCode=0
Jul 17 15:42:40.566898 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.566450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerDied","Data":"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"}
Jul 17 15:42:40.566898 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.566875    2566 scope.go:117] "RemoveContainer" containerID="5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"
Jul 17 15:42:40.567868 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.567841    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:40.567947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.567934    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:40.567996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.567971    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:42:40.753298 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.753259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:40.753823 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:40.753806    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:40.754082 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:40.754034    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:42:41.573222 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:41.573188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerStarted","Data":"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"}
Jul 17 15:42:41.925123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:41.925020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:41.926113 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:41.926083    2566 patch_prober.go:28] interesting pod/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 17 15:42:41.926239 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:41.926139    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 17 15:42:42.925157 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:42.925124    2566 patch_prober.go:28] interesting pod/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 17 15:42:42.925659 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:42.925198    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 17 15:42:43.924863 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:43.924822    2566 patch_prober.go:28] interesting pod/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 17 15:42:43.925087 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:43.924884    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 17 15:42:44.925027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:44.924996    2566 patch_prober.go:28] interesting pod/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 17 15:42:44.925450 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:44.925047    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 17 15:42:45.924607 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:45.924570    2566 patch_prober.go:28] interesting pod/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 17 15:42:45.924803 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:45.924640    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 17 15:42:46.476418 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:46.476380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:46.929045 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:46.929015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:46.930280 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:46.930260    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:42:52.235858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:52.235822    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:53.632401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:53.632364    2566 generic.go:356] "Generic (PLEG): container finished" podID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d" exitCode=1
Jul 17 15:42:53.632832 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:53.632433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"}
Jul 17 15:42:53.632832 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:53.632494    2566 scope.go:117] "RemoveContainer" containerID="f22ff00688362969fb33ab089255a9c9141bcf505704fc5cf6c055a8d72b64eb"
Jul 17 15:42:53.632935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:53.632840    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:42:53.633109 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:53.633086    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:42:56.475911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:56.475870    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:42:56.476361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:42:56.476321    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:42:56.476550 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:42:56.476530    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:00.753845 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:00.753805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:43:00.754378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:00.754252    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:43:00.754487 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:00.754466    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:12.241046 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:12.241015    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:43:12.241466 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:12.241237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:23.235429 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:23.235396    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:43:23.776580 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:23.776542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerStarted","Data":"9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"}
Jul 17 15:43:23.776794 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:23.776775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:43:24.782813 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:24.782776    2566 generic.go:356] "Generic (PLEG): container finished" podID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922" exitCode=1
Jul 17 15:43:24.783316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:24.782833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"}
Jul 17 15:43:24.783316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:24.782880    2566 scope.go:117] "RemoveContainer" containerID="fa183a3ffa5b815580300e3b68236909ed661bd0baf05e25335b5d4112f0680d"
Jul 17 15:43:24.783316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:24.783210    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:43:24.783508 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:24.783460    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:25.789581 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:25.789556    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:43:25.789958 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:25.789752    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:30.753888 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:30.753797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:43:30.754301 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:30.754266    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:43:30.754489 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:30.754464    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:42.235973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:42.235944    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:43:42.236468 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:42.236150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:43:54.235912 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:43:54.235878    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:43:54.236541 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:43:54.236094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:09.236049 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:09.236013    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:44:09.996146 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:09.996113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerStarted","Data":"edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"}
Jul 17 15:44:09.996339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:09.996323    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:44:10.289842 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:10.289755    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:44:10.294867 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:10.294846    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:44:11.002383 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:11.002346    2566 generic.go:356] "Generic (PLEG): container finished" podID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3" exitCode=1
Jul 17 15:44:11.002566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:11.002433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"}
Jul 17 15:44:11.002566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:11.002521    2566 scope.go:117] "RemoveContainer" containerID="9b97ead3cb7875c635e76b392da2fe87c2f56c695ed8e0b76919571e7bbb6922"
Jul 17 15:44:11.002807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:11.002792    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:11.003021 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:44:11.003000    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:12.009459 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:12.009426    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:12.009848 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:44:12.009646    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:20.754016 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:20.753973    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:44:20.754467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:20.754432    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:20.754676 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:44:20.754656    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:34.235855 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:34.235819    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:34.236290 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:44:34.236149    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:49.235306 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:49.235267    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:49.235936 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:44:49.235542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk_odh-ai-gateway-infra(f8a49b21-9ac2-4984-9bff-bd5148d9b60a)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a"
Jul 17 15:44:54.688662 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.688617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:44:54.828120 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.828089    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:44:54.904685 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.904648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") pod \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") "
Jul 17 15:44:54.904836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.904701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmcdm\" (UniqueName: \"kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm\") pod \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") "
Jul 17 15:44:54.904836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.904725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca\") pod \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\" (UID: \"f8a49b21-9ac2-4984-9bff-bd5148d9b60a\") "
Jul 17 15:44:54.905206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.905180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8a49b21-9ac2-4984-9bff-bd5148d9b60a" (UID: "f8a49b21-9ac2-4984-9bff-bd5148d9b60a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:44:54.906903 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.906880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8a49b21-9ac2-4984-9bff-bd5148d9b60a" (UID: "f8a49b21-9ac2-4984-9bff-bd5148d9b60a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:44:54.907814 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:54.907791    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm" (OuterVolumeSpecName: "kube-api-access-xmcdm") pod "f8a49b21-9ac2-4984-9bff-bd5148d9b60a" (UID: "f8a49b21-9ac2-4984-9bff-bd5148d9b60a"). InnerVolumeSpecName "kube-api-access-xmcdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:44:55.005373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.005342    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:44:55.005373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.005367    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcdm\" (UniqueName: \"kubernetes.io/projected/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-kube-api-access-xmcdm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:44:55.005373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.005377    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a49b21-9ac2-4984-9bff-bd5148d9b60a-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:44:55.218416 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.218380    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"
Jul 17 15:44:55.218606 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.218425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk" event={"ID":"f8a49b21-9ac2-4984-9bff-bd5148d9b60a","Type":"ContainerDied","Data":"7ed946c16d2d7b1ff041eaf5d22a6761edd375051b289014ba45c02d508a9cde"}
Jul 17 15:44:55.218606 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.218489    2566 scope.go:117] "RemoveContainer" containerID="edb1ef1a4e811da6981dc6ac295a06948a659f25709fb0d0d67042fb6b1294d3"
Jul 17 15:44:55.240465 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.240437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:44:55.244744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:55.244719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7726776b-cb6ff96d9-mdtgk"]
Jul 17 15:44:56.241036 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:44:56.240991    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" path="/var/lib/kubelet/pods/f8a49b21-9ac2-4984-9bff-bd5148d9b60a/volumes"
Jul 17 15:45:09.617960 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:09.617873    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:45:09.618387 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:09.618106    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy" containerID="cri-o://97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae" gracePeriod=30
Jul 17 15:45:14.878409 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.878378    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:45:14.976414 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9dz9\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976520    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976669    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.976883 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs\") pod \"75382c25-91aa-4815-9451-e3b84e33e9dd\" (UID: \"75382c25-91aa-4815-9451-e3b84e33e9dd\") "
Jul 17 15:45:14.977006 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.976975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:45:14.977350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.977276    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:45:14.977350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.977345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:45:14.977350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.977323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:45:14.977350 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.977420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:45:14.978819 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.978792    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:45:14.978900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.978846    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9" (OuterVolumeSpecName: "kube-api-access-g9dz9") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "kube-api-access-g9dz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:45:14.978900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.978844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:45:14.979332 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:14.979314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "75382c25-91aa-4815-9451-e3b84e33e9dd" (UID: "75382c25-91aa-4815-9451-e3b84e33e9dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:45:15.077777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077738    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077775    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077788    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077797    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dz9\" (UniqueName: \"kubernetes.io/projected/75382c25-91aa-4815-9451-e3b84e33e9dd-kube-api-access-g9dz9\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077807    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077815    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077822    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75382c25-91aa-4815-9451-e3b84e33e9dd-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077830    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75382c25-91aa-4815-9451-e3b84e33e9dd-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.077984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.077838    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75382c25-91aa-4815-9451-e3b84e33e9dd-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:45:15.315147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.315116    2566 generic.go:356] "Generic (PLEG): container finished" podID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerID="97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae" exitCode=0
Jul 17 15:45:15.315308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.315157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerDied","Data":"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"}
Jul 17 15:45:15.315308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.315167    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"
Jul 17 15:45:15.315308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.315198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq" event={"ID":"75382c25-91aa-4815-9451-e3b84e33e9dd","Type":"ContainerDied","Data":"41f118b9459106f24b724396b8b9b401de6a5f599a2a737e5ad31a67db2a503f"}
Jul 17 15:45:15.315308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.315220    2566 scope.go:117] "RemoveContainer" containerID="97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"
Jul 17 15:45:15.324935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.324916    2566 scope.go:117] "RemoveContainer" containerID="5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"
Jul 17 15:45:15.333297 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.333277    2566 scope.go:117] "RemoveContainer" containerID="97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"
Jul 17 15:45:15.333600 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:45:15.333580    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae\": container with ID starting with 97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae not found: ID does not exist" containerID="97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"
Jul 17 15:45:15.333668 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.333610    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae"} err="failed to get container status \"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae\": rpc error: code = NotFound desc = could not find container \"97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae\": container with ID starting with 97588032c26798253ee3f636e81002775ffeffc103bc499645cfdc3cd8058aae not found: ID does not exist"
Jul 17 15:45:15.333668 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.333632    2566 scope.go:117] "RemoveContainer" containerID="5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"
Jul 17 15:45:15.333908 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:45:15.333891    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154\": container with ID starting with 5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154 not found: ID does not exist" containerID="5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"
Jul 17 15:45:15.333967 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.333914    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154"} err="failed to get container status \"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154\": rpc error: code = NotFound desc = could not find container \"5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154\": container with ID starting with 5e6af8d944b5147d71a97a4ea39fe20eb44c8ad8a046a15f752a9e606f497154 not found: ID does not exist"
Jul 17 15:45:15.339905 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.339876    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:45:15.344576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:15.344556    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7726776b-openshift-default-5c75d4b855-nfgrq"]
Jul 17 15:45:16.241425 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:45:16.241389    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" path="/var/lib/kubelet/pods/75382c25-91aa-4815-9451-e3b84e33e9dd/volumes"
Jul 17 15:46:52.225797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.225717    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226131    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226143    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226150    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226194    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226201    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226213    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226218    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226227    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226232    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226241    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226246    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226256    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226260    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226316    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226325    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226330    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226344    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.226341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.226348    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:52.229593 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.229573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.232172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.232144    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 15:46:52.232287 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.232193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-25e005e0-openshift-default-dockercfg-zsnvk\""
Jul 17 15:46:52.261112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.261074    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:46:52.413043 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413251 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413251 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413251 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413506 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.413506 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.413441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47q8\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514197 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47q8\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514682 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514682 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514682 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514910 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.514972 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.515034 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.514983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.515125 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.515039    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.516642 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.516621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.516935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.516916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.526923 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.526890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47q8\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.527792 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.527775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token\") pod \"e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") " pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.541076 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.541034    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:52.704517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.704482    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:46:52.708863 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:46:52.708833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2dd914_5198_4797_bf42_e61ea2b1ec73.slice/crio-ce3a8af9eeea8f9dc0747c5b47699255ee11403308f40b9a8c3458175e0ecf78 WatchSource:0}: Error finding container ce3a8af9eeea8f9dc0747c5b47699255ee11403308f40b9a8c3458175e0ecf78: Status 404 returned error can't find the container with id ce3a8af9eeea8f9dc0747c5b47699255ee11403308f40b9a8c3458175e0ecf78
Jul 17 15:46:52.710994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.710962    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:46:52.711088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.711035    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:46:52.711088 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.711083    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:46:52.755039 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:52.755012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerStarted","Data":"ce3a8af9eeea8f9dc0747c5b47699255ee11403308f40b9a8c3458175e0ecf78"}
Jul 17 15:46:53.761499 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:53.761461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerStarted","Data":"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"}
Jul 17 15:46:53.791046 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:53.790993    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podStartSLOduration=1.790975423 podStartE2EDuration="1.790975423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:46:53.787139716 +0000 UTC m=+2264.062288038" watchObservedRunningTime="2026-07-17 15:46:53.790975423 +0000 UTC m=+2264.066123743"
Jul 17 15:46:54.541936 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:54.541894    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:54.547001 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:54.546972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:54.766758 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:54.766723    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:54.767783 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:54.767764    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:46:56.210181 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210147    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:46:56.210571 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210532    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:56.210571 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:56.210571 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210556    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:56.210571 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210562    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:56.210703 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210611    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e6c36e-bb10-493a-8e2a-bb4bc1dc7385" containerName="istio-proxy"
Jul 17 15:46:56.210703 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210621    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a49b21-9ac2-4984-9bff-bd5148d9b60a" containerName="maas-api"
Jul 17 15:46:56.210703 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.210631    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="75382c25-91aa-4815-9451-e3b84e33e9dd" containerName="istio-proxy"
Jul 17 15:46:56.214385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.214366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.216907 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.216889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25e005e0-serving-cert\""
Jul 17 15:46:56.217174 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.217162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:46:56.225732 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.225700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:46:56.353560 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.353520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.353776 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.353653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.353776 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.353706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb9t\" (UniqueName: \"kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.454993 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.454950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.455204 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.455017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.455204 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.455044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb9t\" (UniqueName: \"kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.455788 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.455767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.457638 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.457619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.468488 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.468425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb9t\" (UniqueName: \"kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t\") pod \"maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.532586 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.532528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:56.673486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.673454    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:46:56.677093 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:46:56.677038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e7c0c9_61ab_45e6_ba63_1bf532bf6865.slice/crio-d8e48dd6cbd91aa421f641be172d99f6f8e1eba6bc16e195346fb2ed3273e544 WatchSource:0}: Error finding container d8e48dd6cbd91aa421f641be172d99f6f8e1eba6bc16e195346fb2ed3273e544: Status 404 returned error can't find the container with id d8e48dd6cbd91aa421f641be172d99f6f8e1eba6bc16e195346fb2ed3273e544
Jul 17 15:46:56.678659 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.678634    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:46:56.776025 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:56.775991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" event={"ID":"15e7c0c9-61ab-45e6-ba63-1bf532bf6865","Type":"ContainerStarted","Data":"d8e48dd6cbd91aa421f641be172d99f6f8e1eba6bc16e195346fb2ed3273e544"}
Jul 17 15:46:57.782525 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:57.782488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" event={"ID":"15e7c0c9-61ab-45e6-ba63-1bf532bf6865","Type":"ContainerStarted","Data":"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"}
Jul 17 15:46:57.782941 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:57.782566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:46:57.801958 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:46:57.801900    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" podStartSLOduration=1.329403866 podStartE2EDuration="1.801883296s" totalImagesPullingTime="472.47943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:46:57.799899008 +0000 UTC m=+2268.075047332" watchObservedRunningTime="2026-07-17 15:46:57.801883296 +0000 UTC m=+2268.077031616"
Jul 17 15:47:03.794366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:03.794331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:47:07.843951 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.843854    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerID="b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3" exitCode=0
Jul 17 15:47:07.843951 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.843937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerDied","Data":"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"}
Jul 17 15:47:07.844452 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.844387    2566 scope.go:117] "RemoveContainer" containerID="b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"
Jul 17 15:47:07.845552 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.845518    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:07.845660 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.845592    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:07.845660 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:07.845627    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:08.851047 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:08.851009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerStarted","Data":"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"}
Jul 17 15:47:09.541824 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:09.541772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:09.542743 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:09.542718    2566 patch_prober.go:28] interesting pod/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Jul 17 15:47:09.542808 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:09.542785    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Jul 17 15:47:09.766934 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:09.766893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:10.542179 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:10.542144    2566 patch_prober.go:28] interesting pod/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Jul 17 15:47:10.542559 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:10.542220    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Jul 17 15:47:11.541499 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:11.541460    2566 patch_prober.go:28] interesting pod/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Jul 17 15:47:11.541695 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:11.541515    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Jul 17 15:47:12.542483 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:12.542441    2566 patch_prober.go:28] interesting pod/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Jul 17 15:47:12.542911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:12.542500    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Jul 17 15:47:13.546674 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:13.546643    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:13.547775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:13.547750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:18.890472 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:18.890439    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:47:18.890929 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:18.890683    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" podUID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" containerName="maas-api" containerID="cri-o://8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28" gracePeriod=30
Jul 17 15:47:19.144687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.144614    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:47:19.275438 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.275404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca\") pod \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") "
Jul 17 15:47:19.275638 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.275501    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggb9t\" (UniqueName: \"kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t\") pod \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") "
Jul 17 15:47:19.275638 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.275534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls\") pod \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\" (UID: \"15e7c0c9-61ab-45e6-ba63-1bf532bf6865\") "
Jul 17 15:47:19.275813 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.275790    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15e7c0c9-61ab-45e6-ba63-1bf532bf6865" (UID: "15e7c0c9-61ab-45e6-ba63-1bf532bf6865"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:47:19.277704 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.277673    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15e7c0c9-61ab-45e6-ba63-1bf532bf6865" (UID: "15e7c0c9-61ab-45e6-ba63-1bf532bf6865"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:47:19.277826 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.277779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t" (OuterVolumeSpecName: "kube-api-access-ggb9t") pod "15e7c0c9-61ab-45e6-ba63-1bf532bf6865" (UID: "15e7c0c9-61ab-45e6-ba63-1bf532bf6865"). InnerVolumeSpecName "kube-api-access-ggb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:47:19.376401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.376360    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb9t\" (UniqueName: \"kubernetes.io/projected/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-kube-api-access-ggb9t\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:19.376401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.376393    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:19.376401 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.376402    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15e7c0c9-61ab-45e6-ba63-1bf532bf6865-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:19.905984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.905938    2566 generic.go:356] "Generic (PLEG): container finished" podID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" containerID="8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28" exitCode=0
Jul 17 15:47:19.905984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.905984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"
Jul 17 15:47:19.906517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.905999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" event={"ID":"15e7c0c9-61ab-45e6-ba63-1bf532bf6865","Type":"ContainerDied","Data":"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"}
Jul 17 15:47:19.906517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.906089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk" event={"ID":"15e7c0c9-61ab-45e6-ba63-1bf532bf6865","Type":"ContainerDied","Data":"d8e48dd6cbd91aa421f641be172d99f6f8e1eba6bc16e195346fb2ed3273e544"}
Jul 17 15:47:19.906517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.906122    2566 scope.go:117] "RemoveContainer" containerID="8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"
Jul 17 15:47:19.915831 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.915814    2566 scope.go:117] "RemoveContainer" containerID="8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"
Jul 17 15:47:19.916168 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:47:19.916145    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28\": container with ID starting with 8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28 not found: ID does not exist" containerID="8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"
Jul 17 15:47:19.916228 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.916179    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28"} err="failed to get container status \"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28\": rpc error: code = NotFound desc = could not find container \"8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28\": container with ID starting with 8bf25d883b7920955f4c7e40040fa39e04a4110a04145deb511141f535a80c28 not found: ID does not exist"
Jul 17 15:47:19.928558 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.928523    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:47:19.932925 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:19.932899    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25e005e0-656b9976d5-rs7gk"]
Jul 17 15:47:20.245261 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:20.243257    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" path="/var/lib/kubelet/pods/15e7c0c9-61ab-45e6-ba63-1bf532bf6865/volumes"
Jul 17 15:47:34.229799 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:34.229760    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:47:34.230428 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:34.229983    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy" containerID="cri-o://373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d" gracePeriod=30
Jul 17 15:47:39.483668 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.483640    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:39.574998 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.574899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.574998 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.574944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m47q8\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.574998 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.574975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575197    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575220    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token\") pod \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\" (UID: \"ec2dd914-5198-4797-bf42-e61ea2b1ec73\") "
Jul 17 15:47:39.575633 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:47:39.575633 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575553    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.575633 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575594    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:47:39.575777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575660    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:47:39.575777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:47:39.576004 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.575980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:47:39.577579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.577551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:47:39.577957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.577927    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:47:39.578037 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.577957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:47:39.578037 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.577979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8" (OuterVolumeSpecName: "kube-api-access-m47q8") pod "ec2dd914-5198-4797-bf42-e61ea2b1ec73" (UID: "ec2dd914-5198-4797-bf42-e61ea2b1ec73"). InnerVolumeSpecName "kube-api-access-m47q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:47:39.675987 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.675950    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.675987 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.675980    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.675987 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.675990    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.676277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.675999    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.676277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.676007    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47q8\" (UniqueName: \"kubernetes.io/projected/ec2dd914-5198-4797-bf42-e61ea2b1ec73-kube-api-access-m47q8\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.676277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.676015    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.676277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.676022    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2dd914-5198-4797-bf42-e61ea2b1ec73-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.676277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.676030    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2dd914-5198-4797-bf42-e61ea2b1ec73-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:47:39.999398 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.999368    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerID="373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d" exitCode=0
Jul 17 15:47:39.999552 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.999421    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"
Jul 17 15:47:39.999552 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.999428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerDied","Data":"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"}
Jul 17 15:47:39.999661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.999597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5" event={"ID":"ec2dd914-5198-4797-bf42-e61ea2b1ec73","Type":"ContainerDied","Data":"ce3a8af9eeea8f9dc0747c5b47699255ee11403308f40b9a8c3458175e0ecf78"}
Jul 17 15:47:39.999661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:39.999630    2566 scope.go:117] "RemoveContainer" containerID="373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"
Jul 17 15:47:40.008877 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.008858    2566 scope.go:117] "RemoveContainer" containerID="b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"
Jul 17 15:47:40.017754 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.017734    2566 scope.go:117] "RemoveContainer" containerID="373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"
Jul 17 15:47:40.018071 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:47:40.018037    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d\": container with ID starting with 373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d not found: ID does not exist" containerID="373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"
Jul 17 15:47:40.018136 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.018084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d"} err="failed to get container status \"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d\": rpc error: code = NotFound desc = could not find container \"373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d\": container with ID starting with 373e0c7628c3ca936c4335d35ae09c0f43d5f45498a139aa8d4eabed967e596d not found: ID does not exist"
Jul 17 15:47:40.018136 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.018104    2566 scope.go:117] "RemoveContainer" containerID="b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"
Jul 17 15:47:40.018348 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:47:40.018330    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3\": container with ID starting with b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3 not found: ID does not exist" containerID="b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"
Jul 17 15:47:40.018392 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.018354    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3"} err="failed to get container status \"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3\": rpc error: code = NotFound desc = could not find container \"b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3\": container with ID starting with b6bc542356860314043737c98c23cd6047da5f0125d623e83634baf2c383e7e3 not found: ID does not exist"
Jul 17 15:47:40.021879 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.021855    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:47:40.026719 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.026696    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-25e005e0-openshift-default-75647fb7dc-x6np5"]
Jul 17 15:47:40.241601 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:40.241569    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" path="/var/lib/kubelet/pods/ec2dd914-5198-4797-bf42-e61ea2b1ec73/volumes"
Jul 17 15:47:55.427791 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.427756    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428146    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" containerName="maas-api"
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428160    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" containerName="maas-api"
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428169    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428182    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:55.428186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428188    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:55.428378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428237    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e7c0c9-61ab-45e6-ba63-1bf532bf6865" containerName="maas-api"
Jul 17 15:47:55.428378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.428246    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:55.430405 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.430387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.432470 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.432454    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bf62e094-openshift-default-dockercfg-kfx7n\""
Jul 17 15:47:55.432861 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.432841    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 15:47:55.442578 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.442549    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:47:55.530565 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530751 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6cmc\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.530922 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.530809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631675 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6cmc\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.631880 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.631941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.632089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.632152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.632339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.632406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.632644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.632627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.634172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.634153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.634474 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.634453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.640975 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.640952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.641259 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.641235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6cmc\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc\") pod \"e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") " pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.740746 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.740715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:55.884530 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.884463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:47:55.886938 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:47:55.886906    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0759dcf4_83d7_40b0_a9e0_7bf8ea696226.slice/crio-b1dc75e6a9e5868d0992265e0f55e27c965446ba4afb24ce9ba21598e4758aba WatchSource:0}: Error finding container b1dc75e6a9e5868d0992265e0f55e27c965446ba4afb24ce9ba21598e4758aba: Status 404 returned error can't find the container with id b1dc75e6a9e5868d0992265e0f55e27c965446ba4afb24ce9ba21598e4758aba
Jul 17 15:47:55.889178 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.889144    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:55.889295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.889225    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:55.889295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:55.889270    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:47:56.077825 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:56.077730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerStarted","Data":"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"}
Jul 17 15:47:56.077825 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:56.077803    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerStarted","Data":"b1dc75e6a9e5868d0992265e0f55e27c965446ba4afb24ce9ba21598e4758aba"}
Jul 17 15:47:56.741235 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:56.741196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:56.746030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:56.746004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:56.768782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:56.768725    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podStartSLOduration=1.768708325 podStartE2EDuration="1.768708325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:47:56.104708297 +0000 UTC m=+2326.379856645" watchObservedRunningTime="2026-07-17 15:47:56.768708325 +0000 UTC m=+2327.043856646"
Jul 17 15:47:57.083610 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:57.083533    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:57.084725 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:57.084707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:47:58.200030 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.199995    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:47:58.200526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.200478    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2dd914-5198-4797-bf42-e61ea2b1ec73" containerName="istio-proxy"
Jul 17 15:47:58.202500 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.202474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.204439 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.204415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bf62e094-serving-cert\""
Jul 17 15:47:58.204573 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.204498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:47:58.215504 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.215478    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:47:58.356172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.356125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.356365 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.356192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5r6\" (UniqueName: \"kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.356365 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.356313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.457150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.457043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5r6\" (UniqueName: \"kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.457150 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.457148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.457341 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.457197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.457779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.457762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.459687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.459667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.467593 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.467569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5r6\" (UniqueName: \"kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6\") pod \"maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.514499 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.514460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:47:58.651887 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:58.651855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:47:58.654334 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:47:58.654309    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd629e8a3_675a_4d5e_9924_41aa111e7b43.slice/crio-381633b450ad7886e1b274cc6c89f994f986c4a4ac9924a12769382428e93316 WatchSource:0}: Error finding container 381633b450ad7886e1b274cc6c89f994f986c4a4ac9924a12769382428e93316: Status 404 returned error can't find the container with id 381633b450ad7886e1b274cc6c89f994f986c4a4ac9924a12769382428e93316
Jul 17 15:47:59.101009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:59.100969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" event={"ID":"d629e8a3-675a-4d5e-9924-41aa111e7b43","Type":"ContainerStarted","Data":"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"}
Jul 17 15:47:59.101182 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:59.101039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" event={"ID":"d629e8a3-675a-4d5e-9924-41aa111e7b43","Type":"ContainerStarted","Data":"381633b450ad7886e1b274cc6c89f994f986c4a4ac9924a12769382428e93316"}
Jul 17 15:47:59.134492 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:47:59.134425    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" podStartSLOduration=0.764309537 podStartE2EDuration="1.13440762s" totalImagesPullingTime="370.098083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:47:59.133047513 +0000 UTC m=+2329.408195857" watchObservedRunningTime="2026-07-17 15:47:59.13440762 +0000 UTC m=+2329.409555941"
Jul 17 15:48:00.105678 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:00.105637    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:48:06.005802 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.005762    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:48:06.008442 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.008416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.010733 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.010704    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bf62e094\"/\"openshift-service-ca.crt\""
Jul 17 15:48:06.011384 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.011328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bf62e094\"/\"default-dockercfg-rn564\""
Jul 17 15:48:06.011384 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.011332    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bf62e094\"/\"kube-root-ca.crt\""
Jul 17 15:48:06.011646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.011402    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bf62e094\"/\"e2e-lifecycle-model-bf62e094-kserve-self-signed-certs\""
Jul 17 15:48:06.019149 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.019121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:48:06.115933 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.115905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:48:06.132996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.132762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.132996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.132862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.132996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.132909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.132996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.132932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnqt\" (UniqueName: \"kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.133382 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.133091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.133382 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.133146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.233805 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.233767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.233996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.233821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.233996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.233889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.233996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.233947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.233996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.233972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnqt\" (UniqueName: \"kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.234331 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.234304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.234516 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.234053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.234600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.234347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.234600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.234375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.236560 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.236539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.236560 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.236553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.243817 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.243793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnqt\" (UniqueName: \"kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt\") pod \"e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") " pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.318975 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.318882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:06.462847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:06.462679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:48:06.465298 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:48:06.465264    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cdeb87f_e61e_4066_a703_1062ef95ce49.slice/crio-4500fc6b22792debc92c1b345ec54642d4122e280dd682180efac5c26a3bbe27 WatchSource:0}: Error finding container 4500fc6b22792debc92c1b345ec54642d4122e280dd682180efac5c26a3bbe27: Status 404 returned error can't find the container with id 4500fc6b22792debc92c1b345ec54642d4122e280dd682180efac5c26a3bbe27
Jul 17 15:48:07.138136 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:07.138100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" event={"ID":"9cdeb87f-e61e-4066-a703-1062ef95ce49","Type":"ContainerStarted","Data":"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"}
Jul 17 15:48:07.138584 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:07.138145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" event={"ID":"9cdeb87f-e61e-4066-a703-1062ef95ce49","Type":"ContainerStarted","Data":"4500fc6b22792debc92c1b345ec54642d4122e280dd682180efac5c26a3bbe27"}
Jul 17 15:48:07.159271 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:07.159213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" podStartSLOduration=2.159177426 podStartE2EDuration="2.159177426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:48:07.158133033 +0000 UTC m=+2337.433281353" watchObservedRunningTime="2026-07-17 15:48:07.159177426 +0000 UTC m=+2337.434325752"
Jul 17 15:48:13.084295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:13.084252    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 15:48:13.084759 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:13.084326    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 15:48:14.172803 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.172762    2566 generic.go:356] "Generic (PLEG): container finished" podID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerID="9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed" exitCode=0
Jul 17 15:48:14.173332 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.172837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerDied","Data":"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"}
Jul 17 15:48:14.173332 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.173306    2566 scope.go:117] "RemoveContainer" containerID="9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"
Jul 17 15:48:14.174304 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.174276    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:48:14.174416 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.174344    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:48:14.174416 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:14.174375    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:48:15.180673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:15.180634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerStarted","Data":"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"}
Jul 17 15:48:15.741548 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:15.741510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:48:15.741781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:15.741760    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 17 15:48:15.741837 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:15.741819    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 17 15:48:16.319104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:16.319069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:16.319563 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:16.319130    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:16.332241 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:16.332207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:16.742022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:16.741986    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 17 15:48:16.742232 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:16.742087    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 17 15:48:17.226285 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:17.226199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:48:17.742171 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:17.742134    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 17 15:48:17.742582 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:17.742194    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 17 15:48:18.741378 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:18.741336    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 17 15:48:18.741583 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:18.741395    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 17 15:48:19.745511 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:19.745481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:48:19.745934 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:19.745750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:48:19.746670 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:19.746652    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:48:35.861487 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:35.861447    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:48:35.861949 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:35.861676    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" podUID="d629e8a3-675a-4d5e-9924-41aa111e7b43" containerName="maas-api" containerID="cri-o://d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f" gracePeriod=30
Jul 17 15:48:36.105238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.105213    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:48:36.230255 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.230162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx5r6\" (UniqueName: \"kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6\") pod \"d629e8a3-675a-4d5e-9924-41aa111e7b43\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") "
Jul 17 15:48:36.230255 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.230215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca\") pod \"d629e8a3-675a-4d5e-9924-41aa111e7b43\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") "
Jul 17 15:48:36.230443 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.230272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls\") pod \"d629e8a3-675a-4d5e-9924-41aa111e7b43\" (UID: \"d629e8a3-675a-4d5e-9924-41aa111e7b43\") "
Jul 17 15:48:36.230598 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.230563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d629e8a3-675a-4d5e-9924-41aa111e7b43" (UID: "d629e8a3-675a-4d5e-9924-41aa111e7b43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:48:36.232528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.232503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d629e8a3-675a-4d5e-9924-41aa111e7b43" (UID: "d629e8a3-675a-4d5e-9924-41aa111e7b43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:48:36.232628 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.232562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6" (OuterVolumeSpecName: "kube-api-access-xx5r6") pod "d629e8a3-675a-4d5e-9924-41aa111e7b43" (UID: "d629e8a3-675a-4d5e-9924-41aa111e7b43"). InnerVolumeSpecName "kube-api-access-xx5r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:48:36.284359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.284326    2566 generic.go:356] "Generic (PLEG): container finished" podID="d629e8a3-675a-4d5e-9924-41aa111e7b43" containerID="d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f" exitCode=0
Jul 17 15:48:36.284575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.284366    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"
Jul 17 15:48:36.284575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.284413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" event={"ID":"d629e8a3-675a-4d5e-9924-41aa111e7b43","Type":"ContainerDied","Data":"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"}
Jul 17 15:48:36.284575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.284469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d" event={"ID":"d629e8a3-675a-4d5e-9924-41aa111e7b43","Type":"ContainerDied","Data":"381633b450ad7886e1b274cc6c89f994f986c4a4ac9924a12769382428e93316"}
Jul 17 15:48:36.284575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.284498    2566 scope.go:117] "RemoveContainer" containerID="d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"
Jul 17 15:48:36.293671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.293653    2566 scope.go:117] "RemoveContainer" containerID="d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"
Jul 17 15:48:36.293940 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:48:36.293918    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f\": container with ID starting with d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f not found: ID does not exist" containerID="d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"
Jul 17 15:48:36.294013 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.293954    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f"} err="failed to get container status \"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f\": rpc error: code = NotFound desc = could not find container \"d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f\": container with ID starting with d88b1033ad7afbadd63973fc51ca4f350a0aa3cb6ffc7c2ebdd32f9f39d4b96f not found: ID does not exist"
Jul 17 15:48:36.301436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.301401    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:48:36.308545 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.308522    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bf62e094-78bc6f8567-c6k4d"]
Jul 17 15:48:36.331289 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.331262    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d629e8a3-675a-4d5e-9924-41aa111e7b43-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:48:36.331289 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.331288    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d629e8a3-675a-4d5e-9924-41aa111e7b43-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:48:36.331289 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:36.331297    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5r6\" (UniqueName: \"kubernetes.io/projected/d629e8a3-675a-4d5e-9924-41aa111e7b43-kube-api-access-xx5r6\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:48:38.241067 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:38.241034    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d629e8a3-675a-4d5e-9924-41aa111e7b43" path="/var/lib/kubelet/pods/d629e8a3-675a-4d5e-9924-41aa111e7b43/volumes"
Jul 17 15:48:45.329506 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:48:45.329471    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:49:00.350502 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.350438    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" podUID="9cdeb87f-e61e-4066-a703-1062ef95ce49" containerName="main" containerID="cri-o://cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05" gracePeriod=45
Jul 17 15:49:00.618646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.618611    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:49:00.652245 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652214    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652269    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652319    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrnqt\" (UniqueName: \"kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652441 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652678 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652452    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home\") pod \"9cdeb87f-e61e-4066-a703-1062ef95ce49\" (UID: \"9cdeb87f-e61e-4066-a703-1062ef95ce49\") "
Jul 17 15:49:00.652678 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652576    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:00.652782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:00.652836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home" (OuterVolumeSpecName: "home") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:00.652898 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.652832    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-tmp-dir\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:00.654929 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.654899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:49:00.655112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.654984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt" (OuterVolumeSpecName: "kube-api-access-hrnqt") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "kube-api-access-hrnqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:49:00.655449 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.655421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm" (OuterVolumeSpecName: "dshm") pod "9cdeb87f-e61e-4066-a703-1062ef95ce49" (UID: "9cdeb87f-e61e-4066-a703-1062ef95ce49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:00.753846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.753811    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-model-cache\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:00.753846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.753840    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-dshm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:00.753846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.753849    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cdeb87f-e61e-4066-a703-1062ef95ce49-home\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:00.753846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.753857    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cdeb87f-e61e-4066-a703-1062ef95ce49-tls-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:00.754153 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:00.753866    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnqt\" (UniqueName: \"kubernetes.io/projected/9cdeb87f-e61e-4066-a703-1062ef95ce49-kube-api-access-hrnqt\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:01.404230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.404194    2566 generic.go:356] "Generic (PLEG): container finished" podID="9cdeb87f-e61e-4066-a703-1062ef95ce49" containerID="cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05" exitCode=0
Jul 17 15:49:01.404635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.404257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"
Jul 17 15:49:01.404635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.404280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" event={"ID":"9cdeb87f-e61e-4066-a703-1062ef95ce49","Type":"ContainerDied","Data":"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"}
Jul 17 15:49:01.404635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.404323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq" event={"ID":"9cdeb87f-e61e-4066-a703-1062ef95ce49","Type":"ContainerDied","Data":"4500fc6b22792debc92c1b345ec54642d4122e280dd682180efac5c26a3bbe27"}
Jul 17 15:49:01.404635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.404340    2566 scope.go:117] "RemoveContainer" containerID="cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"
Jul 17 15:49:01.413480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.413464    2566 scope.go:117] "RemoveContainer" containerID="cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"
Jul 17 15:49:01.413778 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:49:01.413751    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05\": container with ID starting with cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05 not found: ID does not exist" containerID="cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"
Jul 17 15:49:01.413823 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.413788    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05"} err="failed to get container status \"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05\": rpc error: code = NotFound desc = could not find container \"cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05\": container with ID starting with cc98a0269c8ead0269ecb31589e93596205d5206ca7aa25f7a20068f535c4f05 not found: ID does not exist"
Jul 17 15:49:01.427225 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.427190    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:49:01.435888 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:01.435857    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bf62e094/e2e-lifecycle-model-bf62e094-kserve-d8cf57787-n9zlq"]
Jul 17 15:49:02.241495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:02.241462    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cdeb87f-e61e-4066-a703-1062ef95ce49" path="/var/lib/kubelet/pods/9cdeb87f-e61e-4066-a703-1062ef95ce49/volumes"
Jul 17 15:49:10.318475 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:10.318449    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:49:10.324629 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:10.324607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:49:32.558306 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:32.558211    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:49:32.558884 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:32.558511    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" containerID="cri-o://a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b" gracePeriod=30
Jul 17 15:49:34.748283 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:34.748241    2566 patch_prober.go:28] interesting pod/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:49:34.748672 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:34.748325    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:49:37.806013 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.805980    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:49:37.902308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902212    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6cmc\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket\") pod \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\" (UID: \"0759dcf4-83d7-40b0-a9e0-7bf8ea696226\") "
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:37.902671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:37.903217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902882    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:37.903217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902901    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:37.903217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.902913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:37.903217 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.903011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:49:37.903508 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.903377    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data" (OuterVolumeSpecName: "istio-data") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:37.904723 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.904690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc" (OuterVolumeSpecName: "kube-api-access-b6cmc") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "kube-api-access-b6cmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:49:37.904723 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.904693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:49:37.905162 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.905140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token" (OuterVolumeSpecName: "istio-token") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:49:37.905257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:37.905209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0759dcf4-83d7-40b0-a9e0-7bf8ea696226" (UID: "0759dcf4-83d7-40b0-a9e0-7bf8ea696226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:49:38.003385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003342    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6cmc\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-kube-api-access-b6cmc\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003380    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003389    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003396    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003405    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003419    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.003671 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.003428    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0759dcf4-83d7-40b0-a9e0-7bf8ea696226-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:49:38.580078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.579966    2566 generic.go:356] "Generic (PLEG): container finished" podID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerID="a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b" exitCode=0
Jul 17 15:49:38.580078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.580045    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"
Jul 17 15:49:38.580078 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.580052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerDied","Data":"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"}
Jul 17 15:49:38.580319 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.580160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4" event={"ID":"0759dcf4-83d7-40b0-a9e0-7bf8ea696226","Type":"ContainerDied","Data":"b1dc75e6a9e5868d0992265e0f55e27c965446ba4afb24ce9ba21598e4758aba"}
Jul 17 15:49:38.580319 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.580185    2566 scope.go:117] "RemoveContainer" containerID="a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"
Jul 17 15:49:38.589147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.589123    2566 scope.go:117] "RemoveContainer" containerID="9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"
Jul 17 15:49:38.597563 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.597544    2566 scope.go:117] "RemoveContainer" containerID="a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"
Jul 17 15:49:38.597890 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:49:38.597868    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b\": container with ID starting with a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b not found: ID does not exist" containerID="a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"
Jul 17 15:49:38.597932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.597903    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b"} err="failed to get container status \"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b\": rpc error: code = NotFound desc = could not find container \"a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b\": container with ID starting with a217783ab049037846a99d683a1431853c0fe3d4232027e837acfd0469277a2b not found: ID does not exist"
Jul 17 15:49:38.597932 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.597922    2566 scope.go:117] "RemoveContainer" containerID="9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"
Jul 17 15:49:38.598208 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:49:38.598192    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed\": container with ID starting with 9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed not found: ID does not exist" containerID="9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"
Jul 17 15:49:38.598264 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.598214    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed"} err="failed to get container status \"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed\": rpc error: code = NotFound desc = could not find container \"9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed\": container with ID starting with 9d50ef9508fc30e50c1228dfb5b8e986803f58a91e4b79c2dba5fc0fc14b49ed not found: ID does not exist"
Jul 17 15:49:38.600876 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.600850    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:49:38.609878 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:38.609846    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bf62e094-openshift-default-6fcb6c7fd7-8zwq4"]
Jul 17 15:49:40.240639 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:49:40.240600    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" path="/var/lib/kubelet/pods/0759dcf4-83d7-40b0-a9e0-7bf8ea696226/volumes"
Jul 17 15:50:29.302242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302209    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cdeb87f-e61e-4066-a703-1062ef95ce49" containerName="main"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cdeb87f-e61e-4066-a703-1062ef95ce49" containerName="main"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302596    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d629e8a3-675a-4d5e-9924-41aa111e7b43" containerName="maas-api"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302602    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d629e8a3-675a-4d5e-9924-41aa111e7b43" containerName="maas-api"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302621    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302682    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d629e8a3-675a-4d5e-9924-41aa111e7b43" containerName="maas-api"
Jul 17 15:50:29.302687 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302691    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cdeb87f-e61e-4066-a703-1062ef95ce49" containerName="main"
Jul 17 15:50:29.302964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302699    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:29.302964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.302706    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:29.305730 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.305711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.307979 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.307960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f93829-openshift-default-dockercfg-q2zzd\""
Jul 17 15:50:29.308075 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.307964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 15:50:29.319113 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.319083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:50:29.367774 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtdc\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.367774 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368019 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.367971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.368026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.368247 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.368049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469377 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469377 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtdc\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.469661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.470020 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.470020 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.470020 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.469971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.470210 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.470104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.470353 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.470317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.471968 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.471939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.472125 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.472024    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.478289 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.478264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.478428 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.478393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtdc\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc\") pod \"e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") " pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.616224 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.616119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:29.760870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.760830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:50:29.763961 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:50:29.763930    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf5334a_7a43_4cca_a8d7_f20d626bb72d.slice/crio-c0ca8a6fffee51cd140727fab8b690836faa3e237454e8100718b66e08be32d8 WatchSource:0}: Error finding container c0ca8a6fffee51cd140727fab8b690836faa3e237454e8100718b66e08be32d8: Status 404 returned error can't find the container with id c0ca8a6fffee51cd140727fab8b690836faa3e237454e8100718b66e08be32d8
Jul 17 15:50:29.765991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.765955    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:29.766107 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.766029    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:29.766107 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.766088    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:29.822133 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:29.822051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerStarted","Data":"c0ca8a6fffee51cd140727fab8b690836faa3e237454e8100718b66e08be32d8"}
Jul 17 15:50:30.827991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:30.827952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerStarted","Data":"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"}
Jul 17 15:50:30.850084 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:30.850013    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podStartSLOduration=1.84999623 podStartE2EDuration="1.84999623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:50:30.84861836 +0000 UTC m=+2481.123766680" watchObservedRunningTime="2026-07-17 15:50:30.84999623 +0000 UTC m=+2481.125144609"
Jul 17 15:50:31.617170 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:31.617129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:31.622163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:31.622133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:31.832530 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:31.832496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:31.833517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:31.833497    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:32.944274 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.944233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:50:32.944686 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.944595    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:32.944686 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.944606    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0759dcf4-83d7-40b0-a9e0-7bf8ea696226" containerName="istio-proxy"
Jul 17 15:50:32.948194 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.948176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:32.950180 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.950151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f93829-serving-cert\""
Jul 17 15:50:32.950319 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.950219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:50:32.957909 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:32.957877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:50:33.006807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.006772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ks2\" (UniqueName: \"kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.006807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.006811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.007043 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.006831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.107358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.107322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ks2\" (UniqueName: \"kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.107597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.107365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.107597 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.107392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.107597 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:50:33.107493    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-serving-cert: secret "maas-api-e2e-inf-a-f93829-serving-cert" not found
Jul 17 15:50:33.107597 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:50:33.107581    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls podName:f168db41-deb2-4979-ae1e-a42e89912215 nodeName:}" failed. No retries permitted until 2026-07-17 15:50:33.607554479 +0000 UTC m=+2483.882702778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls") pod "maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" (UID: "f168db41-deb2-4979-ae1e-a42e89912215") : secret "maas-api-e2e-inf-a-f93829-serving-cert" not found
Jul 17 15:50:33.108205 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.108179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.123135 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.123099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ks2\" (UniqueName: \"kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.613129 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.613053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.615697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.615672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") pod \"maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:33.861312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:33.861274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:34.000986 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:34.000961    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:50:34.003051 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:50:34.003021    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf168db41_deb2_4979_ae1e_a42e89912215.slice/crio-3eaa5cf141ddfd0ca68bbbb1cb32e015f05540061ebcf0e0ef92c98d1e015cbe WatchSource:0}: Error finding container 3eaa5cf141ddfd0ca68bbbb1cb32e015f05540061ebcf0e0ef92c98d1e015cbe: Status 404 returned error can't find the container with id 3eaa5cf141ddfd0ca68bbbb1cb32e015f05540061ebcf0e0ef92c98d1e015cbe
Jul 17 15:50:34.847386 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:34.847347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" event={"ID":"f168db41-deb2-4979-ae1e-a42e89912215","Type":"ContainerStarted","Data":"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"}
Jul 17 15:50:34.847601 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:34.847402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" event={"ID":"f168db41-deb2-4979-ae1e-a42e89912215","Type":"ContainerStarted","Data":"3eaa5cf141ddfd0ca68bbbb1cb32e015f05540061ebcf0e0ef92c98d1e015cbe"}
Jul 17 15:50:34.847601 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:34.847467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:34.866127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:34.866049    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" podStartSLOduration=2.23186319 podStartE2EDuration="2.866031897s" totalImagesPullingTime="634.168707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:50:34.864596904 +0000 UTC m=+2485.139745225" watchObservedRunningTime="2026-07-17 15:50:34.866031897 +0000 UTC m=+2485.141180252"
Jul 17 15:50:37.908277 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.908242    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:50:37.911931 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.911908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.914283 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.914263    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1e887c-openshift-default-dockercfg-g6kxc\""
Jul 17 15:50:37.924901 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.924875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:50:37.956683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956683 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rv5j\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:37.956919 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:37.956882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.057990 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.057946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.057990 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.057991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rv5j\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058486    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.058842 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.058716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.059079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.059035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.060646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.060626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.061109 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.061089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.069859 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.069831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rv5j\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.072215 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.072194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token\") pod \"e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") " pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.221709 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.221603    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:38.365051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.365024    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:50:38.367673 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:50:38.367630    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0727d83e_8226_41a4_870d_597936d9a450.slice/crio-99b25973a3eaeb02df19447dafca45242cf055fdca7a0eb536c3f8fbaec8ddf6 WatchSource:0}: Error finding container 99b25973a3eaeb02df19447dafca45242cf055fdca7a0eb536c3f8fbaec8ddf6: Status 404 returned error can't find the container with id 99b25973a3eaeb02df19447dafca45242cf055fdca7a0eb536c3f8fbaec8ddf6
Jul 17 15:50:38.369811 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.369777    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:38.369923 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.369858    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:38.369923 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.369902    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:38.871199 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.871163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerStarted","Data":"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"}
Jul 17 15:50:38.871394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.871228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerStarted","Data":"99b25973a3eaeb02df19447dafca45242cf055fdca7a0eb536c3f8fbaec8ddf6"}
Jul 17 15:50:38.894985 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:38.894931    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podStartSLOduration=1.894912597 podStartE2EDuration="1.894912597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:50:38.893000348 +0000 UTC m=+2489.168148681" watchObservedRunningTime="2026-07-17 15:50:38.894912597 +0000 UTC m=+2489.170060918"
Jul 17 15:50:39.222126 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:39.221990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:39.227098 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:39.227041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:39.875429 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:39.875387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:39.876397 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:39.876377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:40.852496 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.852450    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:50:40.856052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.856026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.858104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.858043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1e887c-serving-cert\""
Jul 17 15:50:40.858957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.858937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:50:40.867294 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.867248    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:50:40.892127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.892092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.892396 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.892367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b988m\" (UniqueName: \"kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.892564 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.892542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.993968 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.993921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.993968 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.993973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b988m\" (UniqueName: \"kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.994273 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.994117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.994853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.994767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:40.996728 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:40.996704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:41.003794 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.003764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b988m\" (UniqueName: \"kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m\") pod \"maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:41.167818 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.167721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:41.312716 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.312688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:50:41.314515 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:50:41.314487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43f8fb6_ddb4_483c_87ae_360cec82771d.slice/crio-c7b3487eafa104e02cf5ff75d8c3bd7f01ad3f82c9ba23a3d639a5f56f2be47a WatchSource:0}: Error finding container c7b3487eafa104e02cf5ff75d8c3bd7f01ad3f82c9ba23a3d639a5f56f2be47a: Status 404 returned error can't find the container with id c7b3487eafa104e02cf5ff75d8c3bd7f01ad3f82c9ba23a3d639a5f56f2be47a
Jul 17 15:50:41.887035 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.886991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" event={"ID":"c43f8fb6-ddb4-483c-87ae-360cec82771d","Type":"ContainerStarted","Data":"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"}
Jul 17 15:50:41.887505 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.887053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" event={"ID":"c43f8fb6-ddb4-483c-87ae-360cec82771d","Type":"ContainerStarted","Data":"c7b3487eafa104e02cf5ff75d8c3bd7f01ad3f82c9ba23a3d639a5f56f2be47a"}
Jul 17 15:50:41.887505 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:41.887113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:47.874244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:47.874194    2566 patch_prober.go:28] interesting pod/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 15:50:47.874648 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:47.874251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 15:50:47.904844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:47.904798    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:50:47.924872 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:47.924815    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" podStartSLOduration=7.582843828 podStartE2EDuration="7.924798245s" totalImagesPullingTime="341.954417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:50:41.908652104 +0000 UTC m=+2492.183800440" watchObservedRunningTime="2026-07-17 15:50:47.924798245 +0000 UTC m=+2498.199946566"
Jul 17 15:50:48.930295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.930214    2566 generic.go:356] "Generic (PLEG): container finished" podID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerID="cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b" exitCode=0
Jul 17 15:50:48.930295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.930263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerDied","Data":"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"}
Jul 17 15:50:48.930760 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.930683    2566 scope.go:117] "RemoveContainer" containerID="cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"
Jul 17 15:50:48.931736 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.931704    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:48.931828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.931781    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:48.931828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:48.931811    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:49.938862 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:49.938822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerStarted","Data":"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"}
Jul 17 15:50:50.616602 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:50.616558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:50.617038 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:50.617006    2566 patch_prober.go:28] interesting pod/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 17 15:50:50.617171 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:50.617090    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 17 15:50:51.617572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:51.617526    2566 patch_prober.go:28] interesting pod/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 17 15:50:51.618104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:51.617617    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 17 15:50:52.617431 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:52.617384    2566 patch_prober.go:28] interesting pod/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 17 15:50:52.617626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:52.617462    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 17 15:50:53.617149 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:53.616738    2566 patch_prober.go:28] interesting pod/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 17 15:50:53.617149 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:53.616811    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 17 15:50:54.620104 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:54.620039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:54.620613 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:54.620414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:54.621279 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:54.621259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:50:55.020118 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.020083    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:58792->10.134.0.47:15021: read: connection reset by peer" start-of-body=
Jul 17 15:50:55.020284 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.020162    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:58792->10.134.0.47:15021: read: connection reset by peer"
Jul 17 15:50:55.972202 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.972170    2566 generic.go:356] "Generic (PLEG): container finished" podID="0727d83e-8226-41a4-870d-597936d9a450" containerID="e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6" exitCode=0
Jul 17 15:50:55.972611 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.972244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerDied","Data":"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"}
Jul 17 15:50:55.972825 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.972807    2566 scope.go:117] "RemoveContainer" containerID="e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"
Jul 17 15:50:55.974144 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.973859    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:55.974144 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.973956    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:55.974144 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:55.974016    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:50:56.980090 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:56.980037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerStarted","Data":"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"}
Jul 17 15:50:57.222049 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.222006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:50:57.222331 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.222305    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 17 15:50:57.222426 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.222373    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 17 15:50:57.301471 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.301432    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:50:57.322363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.322315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:50:57.322541 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.322443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.325486 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.325451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f93829\"/\"test-model-f93829-kserve-self-signed-certs\""
Jul 17 15:50:57.325662 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.325546    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f93829\"/\"kube-root-ca.crt\""
Jul 17 15:50:57.325792 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.325771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f93829\"/\"default-dockercfg-v8j52\""
Jul 17 15:50:57.326148 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.325780    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f93829\"/\"openshift-service-ca.crt\""
Jul 17 15:50:57.474656 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.474857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.474857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.474857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.474964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffsj\" (UniqueName: \"kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.474964 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.474914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576468 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576468 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffsj\" (UniqueName: \"kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.576918 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.576907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.578996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.578970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.579429 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.579408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.585767 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.585745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffsj\" (UniqueName: \"kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj\") pod \"test-model-f93829-kserve-55fb55b875-j2q5j\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") " pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.633235 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.633195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:50:57.817194 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.810637    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:50:57.821389 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:50:57.821346    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad3f5d9_8081_406a_a2f9_69fa264e6957.slice/crio-5dc01f51788156a41454e75d27531385d1b67166697ae233446c0a66ccae71d4 WatchSource:0}: Error finding container 5dc01f51788156a41454e75d27531385d1b67166697ae233446c0a66ccae71d4: Status 404 returned error can't find the container with id 5dc01f51788156a41454e75d27531385d1b67166697ae233446c0a66ccae71d4
Jul 17 15:50:57.985829 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.985790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" event={"ID":"aad3f5d9-8081-406a-a2f9-69fa264e6957","Type":"ContainerStarted","Data":"e0aba2f1298ca55ef09dd6ba871fab8d19a846e5ee10b0d18d77d2d50aa5c552"}
Jul 17 15:50:57.986230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:57.985837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" event={"ID":"aad3f5d9-8081-406a-a2f9-69fa264e6957","Type":"ContainerStarted","Data":"5dc01f51788156a41454e75d27531385d1b67166697ae233446c0a66ccae71d4"}
Jul 17 15:50:58.004572 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:58.004498    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" podStartSLOduration=1.004472589 podStartE2EDuration="1.004472589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:50:58.003574882 +0000 UTC m=+2508.278723203" watchObservedRunningTime="2026-07-17 15:50:58.004472589 +0000 UTC m=+2508.279620913"
Jul 17 15:50:58.222321 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:58.222219    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 17 15:50:58.222321 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:58.222284    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 17 15:50:59.222289 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:59.222245    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 17 15:50:59.222782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:50:59.222325    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 17 15:51:00.222467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:00.222365    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 17 15:51:00.222467 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:00.222420    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 17 15:51:01.225605 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:01.225572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:51:01.226038 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:01.225851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:51:01.226682 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:01.226664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:51:07.634123 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:07.634087    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:51:07.634550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:07.634139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:51:07.646957 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:07.646932    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:51:08.046529 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:51:08.046500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:52:04.132975 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:04.132928    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:52:15.666457 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:15.666420    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:52:15.666869 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:15.666672    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" podUID="f168db41-deb2-4979-ae1e-a42e89912215" containerName="maas-api" containerID="cri-o://39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b" gracePeriod=30
Jul 17 15:52:15.919413 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:15.919347    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:52:16.057954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.057921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca\") pod \"f168db41-deb2-4979-ae1e-a42e89912215\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") "
Jul 17 15:52:16.057954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.057959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2ks2\" (UniqueName: \"kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2\") pod \"f168db41-deb2-4979-ae1e-a42e89912215\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") "
Jul 17 15:52:16.058272 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.058001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") pod \"f168db41-deb2-4979-ae1e-a42e89912215\" (UID: \"f168db41-deb2-4979-ae1e-a42e89912215\") "
Jul 17 15:52:16.058425 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.058378    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f168db41-deb2-4979-ae1e-a42e89912215" (UID: "f168db41-deb2-4979-ae1e-a42e89912215"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:52:16.060323 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.060295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f168db41-deb2-4979-ae1e-a42e89912215" (UID: "f168db41-deb2-4979-ae1e-a42e89912215"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:52:16.060438 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.060403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2" (OuterVolumeSpecName: "kube-api-access-w2ks2") pod "f168db41-deb2-4979-ae1e-a42e89912215" (UID: "f168db41-deb2-4979-ae1e-a42e89912215"). InnerVolumeSpecName "kube-api-access-w2ks2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:52:16.159544 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.159504    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f168db41-deb2-4979-ae1e-a42e89912215-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:16.159544 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.159536    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2ks2\" (UniqueName: \"kubernetes.io/projected/f168db41-deb2-4979-ae1e-a42e89912215-kube-api-access-w2ks2\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:16.159544 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.159547    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f168db41-deb2-4979-ae1e-a42e89912215-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:16.364183 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.364141    2566 generic.go:356] "Generic (PLEG): container finished" podID="f168db41-deb2-4979-ae1e-a42e89912215" containerID="39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b" exitCode=0
Jul 17 15:52:16.364358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.364206    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"
Jul 17 15:52:16.364358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.364224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" event={"ID":"f168db41-deb2-4979-ae1e-a42e89912215","Type":"ContainerDied","Data":"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"}
Jul 17 15:52:16.364358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.364281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" event={"ID":"f168db41-deb2-4979-ae1e-a42e89912215","Type":"ContainerDied","Data":"3eaa5cf141ddfd0ca68bbbb1cb32e015f05540061ebcf0e0ef92c98d1e015cbe"}
Jul 17 15:52:16.364358 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.364303    2566 scope.go:117] "RemoveContainer" containerID="39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"
Jul 17 15:52:16.373621 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.373603    2566 scope.go:117] "RemoveContainer" containerID="39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"
Jul 17 15:52:16.373902 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:52:16.373873    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b\": container with ID starting with 39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b not found: ID does not exist" containerID="39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"
Jul 17 15:52:16.374040 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.373906    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b"} err="failed to get container status \"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b\": rpc error: code = NotFound desc = could not find container \"39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b\": container with ID starting with 39a03585ea11ce57eed4502fc1ec14d71eed15aefa30be2735c98e348d09de4b not found: ID does not exist"
Jul 17 15:52:16.380807 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.380777    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:52:16.387177 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:16.387151    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2"]
Jul 17 15:52:18.243812 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:18.243778    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f168db41-deb2-4979-ae1e-a42e89912215" path="/var/lib/kubelet/pods/f168db41-deb2-4979-ae1e-a42e89912215/volumes"
Jul 17 15:52:18.853860 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:18.853820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f93829-59d9bf99d-7mmc2" podUID="f168db41-deb2-4979-ae1e-a42e89912215" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 15:52:19.154366 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.154270    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" podUID="aad3f5d9-8081-406a-a2f9-69fa264e6957" containerName="main" containerID="cri-o://e0aba2f1298ca55ef09dd6ba871fab8d19a846e5ee10b0d18d77d2d50aa5c552" gracePeriod=45
Jul 17 15:52:19.381367 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.381339    2566 generic.go:356] "Generic (PLEG): container finished" podID="aad3f5d9-8081-406a-a2f9-69fa264e6957" containerID="e0aba2f1298ca55ef09dd6ba871fab8d19a846e5ee10b0d18d77d2d50aa5c552" exitCode=0
Jul 17 15:52:19.381730 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.381392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" event={"ID":"aad3f5d9-8081-406a-a2f9-69fa264e6957","Type":"ContainerDied","Data":"e0aba2f1298ca55ef09dd6ba871fab8d19a846e5ee10b0d18d77d2d50aa5c552"}
Jul 17 15:52:19.401112 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.401080    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:52:19.488566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ffsj\" (UniqueName: \"kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488682    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488720    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488980 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir\") pod \"aad3f5d9-8081-406a-a2f9-69fa264e6957\" (UID: \"aad3f5d9-8081-406a-a2f9-69fa264e6957\") "
Jul 17 15:52:19.488980 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.488835    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home" (OuterVolumeSpecName: "home") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:19.489127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.489032    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:19.489127 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.489043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache" (OuterVolumeSpecName: "model-cache") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:19.489257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.489153    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-model-cache\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:19.489257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.489166    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-tmp-dir\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:19.489257 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.489174    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-home\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:19.491236 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.491209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj" (OuterVolumeSpecName: "kube-api-access-8ffsj") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "kube-api-access-8ffsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:52:19.491336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.491239    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:52:19.491336 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.491270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm" (OuterVolumeSpecName: "dshm") pod "aad3f5d9-8081-406a-a2f9-69fa264e6957" (UID: "aad3f5d9-8081-406a-a2f9-69fa264e6957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:19.589948 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.589910    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad3f5d9-8081-406a-a2f9-69fa264e6957-tls-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:19.589948 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.589941    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffsj\" (UniqueName: \"kubernetes.io/projected/aad3f5d9-8081-406a-a2f9-69fa264e6957-kube-api-access-8ffsj\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:19.589948 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:19.589956    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad3f5d9-8081-406a-a2f9-69fa264e6957-dshm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:20.387571 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:20.387539    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"
Jul 17 15:52:20.388028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:20.387582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j" event={"ID":"aad3f5d9-8081-406a-a2f9-69fa264e6957","Type":"ContainerDied","Data":"5dc01f51788156a41454e75d27531385d1b67166697ae233446c0a66ccae71d4"}
Jul 17 15:52:20.388028 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:20.387636    2566 scope.go:117] "RemoveContainer" containerID="e0aba2f1298ca55ef09dd6ba871fab8d19a846e5ee10b0d18d77d2d50aa5c552"
Jul 17 15:52:20.419440 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:20.419406    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:52:20.422553 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:20.422528    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f93829/test-model-f93829-kserve-55fb55b875-j2q5j"]
Jul 17 15:52:22.240778 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:22.240739    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad3f5d9-8081-406a-a2f9-69fa264e6957" path="/var/lib/kubelet/pods/aad3f5d9-8081-406a-a2f9-69fa264e6957/volumes"
Jul 17 15:52:30.968132 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:30.968007    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:52:30.968591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:30.968290    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy" containerID="cri-o://f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79" gracePeriod=30
Jul 17 15:52:36.227779 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.227751    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:52:36.342696 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342696 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whtdc\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.342954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.342935    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert\") pod \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\" (UID: \"dbf5334a-7a43-4cca-a8d7-f20d626bb72d\") "
Jul 17 15:52:36.343343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.343013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:36.343343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.343266    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.343343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.343263    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:36.343457 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.343414    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:52:36.343616 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.343588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:36.344173 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.344085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:36.345200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.345167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:52:36.345327 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.345309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc" (OuterVolumeSpecName: "kube-api-access-whtdc") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "kube-api-access-whtdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:52:36.345637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.345602    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:52:36.345729 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.345683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbf5334a-7a43-4cca-a8d7-f20d626bb72d" (UID: "dbf5334a-7a43-4cca-a8d7-f20d626bb72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:52:36.444027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.443973    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444018    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444026    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444035    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444027 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444043    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444464 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444051    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtdc\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-kube-api-access-whtdc\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444464 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444077    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.444464 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.444086    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf5334a-7a43-4cca-a8d7-f20d626bb72d-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:36.468753 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.468714    2566 generic.go:356] "Generic (PLEG): container finished" podID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerID="f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79" exitCode=0
Jul 17 15:52:36.468947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.468786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"
Jul 17 15:52:36.468947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.468782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerDied","Data":"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"}
Jul 17 15:52:36.468947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.468923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2" event={"ID":"dbf5334a-7a43-4cca-a8d7-f20d626bb72d","Type":"ContainerDied","Data":"c0ca8a6fffee51cd140727fab8b690836faa3e237454e8100718b66e08be32d8"}
Jul 17 15:52:36.469135 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.468956    2566 scope.go:117] "RemoveContainer" containerID="f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"
Jul 17 15:52:36.480138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.480112    2566 scope.go:117] "RemoveContainer" containerID="cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"
Jul 17 15:52:36.488582 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.488561    2566 scope.go:117] "RemoveContainer" containerID="f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"
Jul 17 15:52:36.488867 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:52:36.488848    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79\": container with ID starting with f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79 not found: ID does not exist" containerID="f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"
Jul 17 15:52:36.488917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.488878    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79"} err="failed to get container status \"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79\": rpc error: code = NotFound desc = could not find container \"f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79\": container with ID starting with f9236a95c40073f590183a6aec6ac47266e1181459a80377a31267ffa7497f79 not found: ID does not exist"
Jul 17 15:52:36.488917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.488898    2566 scope.go:117] "RemoveContainer" containerID="cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"
Jul 17 15:52:36.489164 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:52:36.489146    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b\": container with ID starting with cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b not found: ID does not exist" containerID="cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"
Jul 17 15:52:36.489221 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.489168    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b"} err="failed to get container status \"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b\": rpc error: code = NotFound desc = could not find container \"cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b\": container with ID starting with cb76290119f5c2b863787ada5c3f8fbc3b85aac639bee511bd3259742fa52f9b not found: ID does not exist"
Jul 17 15:52:36.492584 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.492557    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:52:36.497864 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:36.497840    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f93829-openshift-default-6dcbbdc7f5-stkm2"]
Jul 17 15:52:38.241423 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:38.241393    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" path="/var/lib/kubelet/pods/dbf5334a-7a43-4cca-a8d7-f20d626bb72d/volumes"
Jul 17 15:52:41.976905 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:41.976871    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:52:41.977329 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:41.977151    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" podUID="c43f8fb6-ddb4-483c-87ae-360cec82771d" containerName="maas-api" containerID="cri-o://e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a" gracePeriod=30
Jul 17 15:52:42.226315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.226284    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:52:42.300359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.300320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca\") pod \"c43f8fb6-ddb4-483c-87ae-360cec82771d\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") "
Jul 17 15:52:42.300359 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.300367    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b988m\" (UniqueName: \"kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m\") pod \"c43f8fb6-ddb4-483c-87ae-360cec82771d\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") "
Jul 17 15:52:42.300575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.300394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls\") pod \"c43f8fb6-ddb4-483c-87ae-360cec82771d\" (UID: \"c43f8fb6-ddb4-483c-87ae-360cec82771d\") "
Jul 17 15:52:42.300795 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.300770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c43f8fb6-ddb4-483c-87ae-360cec82771d" (UID: "c43f8fb6-ddb4-483c-87ae-360cec82771d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:52:42.302662 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.302634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c43f8fb6-ddb4-483c-87ae-360cec82771d" (UID: "c43f8fb6-ddb4-483c-87ae-360cec82771d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:52:42.302780 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.302759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m" (OuterVolumeSpecName: "kube-api-access-b988m") pod "c43f8fb6-ddb4-483c-87ae-360cec82771d" (UID: "c43f8fb6-ddb4-483c-87ae-360cec82771d"). InnerVolumeSpecName "kube-api-access-b988m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:52:42.401528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.401483    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43f8fb6-ddb4-483c-87ae-360cec82771d-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:42.401528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.401518    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b988m\" (UniqueName: \"kubernetes.io/projected/c43f8fb6-ddb4-483c-87ae-360cec82771d-kube-api-access-b988m\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:42.401528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.401531    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c43f8fb6-ddb4-483c-87ae-360cec82771d-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:52:42.498792 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.498758    2566 generic.go:356] "Generic (PLEG): container finished" podID="c43f8fb6-ddb4-483c-87ae-360cec82771d" containerID="e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a" exitCode=0
Jul 17 15:52:42.498973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.498800    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"
Jul 17 15:52:42.498973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.498832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" event={"ID":"c43f8fb6-ddb4-483c-87ae-360cec82771d","Type":"ContainerDied","Data":"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"}
Jul 17 15:52:42.498973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.498884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq" event={"ID":"c43f8fb6-ddb4-483c-87ae-360cec82771d","Type":"ContainerDied","Data":"c7b3487eafa104e02cf5ff75d8c3bd7f01ad3f82c9ba23a3d639a5f56f2be47a"}
Jul 17 15:52:42.498973 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.498914    2566 scope.go:117] "RemoveContainer" containerID="e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"
Jul 17 15:52:42.508222 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.508201    2566 scope.go:117] "RemoveContainer" containerID="e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"
Jul 17 15:52:42.508504 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:52:42.508486    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a\": container with ID starting with e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a not found: ID does not exist" containerID="e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"
Jul 17 15:52:42.508550 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.508514    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a"} err="failed to get container status \"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a\": rpc error: code = NotFound desc = could not find container \"e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a\": container with ID starting with e57c678f74ee11a0a43faa6f84ac26dd41383581cf1c1f8d6fd32191c8ff5f0a not found: ID does not exist"
Jul 17 15:52:42.523798 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.523767    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:52:42.529604 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:42.529579    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e887c-5cf9f4c78b-lhlcq"]
Jul 17 15:52:44.240316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:44.240282    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43f8fb6-ddb4-483c-87ae-360cec82771d" path="/var/lib/kubelet/pods/c43f8fb6-ddb4-483c-87ae-360cec82771d/volumes"
Jul 17 15:52:57.102367 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:57.102327    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:52:57.102792 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:57.102543    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" containerID="cri-o://a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23" gracePeriod=30
Jul 17 15:52:58.029680 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.029639    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:52:58.030105 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030087    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43f8fb6-ddb4-483c-87ae-360cec82771d" containerName="maas-api"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43f8fb6-ddb4-483c-87ae-360cec82771d" containerName="maas-api"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030118    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad3f5d9-8081-406a-a2f9-69fa264e6957" containerName="main"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030128    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad3f5d9-8081-406a-a2f9-69fa264e6957" containerName="main"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030140    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030148    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030158    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f168db41-deb2-4979-ae1e-a42e89912215" containerName="maas-api"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030166    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f168db41-deb2-4979-ae1e-a42e89912215" containerName="maas-api"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030184    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030208 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030192    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030295    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43f8fb6-ddb4-483c-87ae-360cec82771d" containerName="maas-api"
Jul 17 15:52:58.030637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030308    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030319    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf5334a-7a43-4cca-a8d7-f20d626bb72d" containerName="istio-proxy"
Jul 17 15:52:58.030637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030330    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f168db41-deb2-4979-ae1e-a42e89912215" containerName="maas-api"
Jul 17 15:52:58.030637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.030338    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad3f5d9-8081-406a-a2f9-69fa264e6957" containerName="main"
Jul 17 15:52:58.033388 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.033366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.037053 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.037030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f54d84-openshift-default-dockercfg-ws2k5\""
Jul 17 15:52:58.054295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.054264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:52:58.152757 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.152757 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.152994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.153112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.153141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.153238 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.153175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzfv\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.253954 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.253917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.253963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.253980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254035    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254196 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzfv\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254490 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254490 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254490 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.254889 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.254871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.256522 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.256498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.257087 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.257051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.262302 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.262270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.262443 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.262340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzfv\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv\") pod \"e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") " pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.343780 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.343692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:58.493114 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.493089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:52:58.504859 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.504820    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:52:58.504982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.504892    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:52:58.504982 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.504919    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:52:58.577739 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.577704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerStarted","Data":"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"}
Jul 17 15:52:58.577942 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.577779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerStarted","Data":"3401fd146de5b335e2ba189cbaf3baa890d7f4c30c7506cd1711aa91b3ce370f"}
Jul 17 15:52:58.604882 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:58.604771    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podStartSLOduration=1.6047479070000001 podStartE2EDuration="1.604747907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:52:58.603245039 +0000 UTC m=+2628.878393361" watchObservedRunningTime="2026-07-17 15:52:58.604747907 +0000 UTC m=+2628.879896228"
Jul 17 15:52:59.344385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:59.344340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:59.349407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:59.349379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:59.582637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:59.582599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:52:59.583581 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:52:59.583559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:00.912124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.912043    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:00.915579 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.915561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:00.917360 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.917338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f54d84-serving-cert\""
Jul 17 15:53:00.917503 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.917361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:53:00.928777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.928749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:00.981343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.981302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:00.981343 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.981349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:00.981584 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:00.981371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcpw\" (UniqueName: \"kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.082031 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.081986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.082031 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.082027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcpw\" (UniqueName: \"kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.082345 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.082219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.082828 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.082804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.084764 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.084745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.092213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.092187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcpw\" (UniqueName: \"kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw\") pod \"maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.226514 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.226419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:01.228637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.228602    2566 patch_prober.go:28] interesting pod/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:53:01.228760 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.228654    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:53:01.382325 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.382156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:01.385159 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:53:01.385131    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8abf17_41e6_4e0d_bc59_43e37d0b86de.slice/crio-5e4475c3cf05661e25a0b6e7d8ae393191b14556863fa23198e6866c6afba984 WatchSource:0}: Error finding container 5e4475c3cf05661e25a0b6e7d8ae393191b14556863fa23198e6866c6afba984: Status 404 returned error can't find the container with id 5e4475c3cf05661e25a0b6e7d8ae393191b14556863fa23198e6866c6afba984
Jul 17 15:53:01.386641 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.386622    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 15:53:01.594874 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:01.594838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" event={"ID":"7c8abf17-41e6-4e0d-bc59-43e37d0b86de","Type":"ContainerStarted","Data":"5e4475c3cf05661e25a0b6e7d8ae393191b14556863fa23198e6866c6afba984"}
Jul 17 15:53:02.368644 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.368617    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:53:02.394840 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.394811    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395006 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.394861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395006 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.394893    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395186 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395147    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rv5j\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395234 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395234 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:53:02.395339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:02.395339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket\") pod \"0727d83e-8226-41a4-870d-597936d9a450\" (UID: \"0727d83e-8226-41a4-870d-597936d9a450\") "
Jul 17 15:53:02.395573 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:02.395752 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395730    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.396146 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.396114    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0727d83e-8226-41a4-870d-597936d9a450-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.396246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.395749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:02.396246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.396134    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.396363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.396314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data" (OuterVolumeSpecName: "istio-data") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:02.397744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.397707    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:53:02.398022 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.398001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:02.398117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.398029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token" (OuterVolumeSpecName: "istio-token") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:02.398444 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.398420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j" (OuterVolumeSpecName: "kube-api-access-4rv5j") pod "0727d83e-8226-41a4-870d-597936d9a450" (UID: "0727d83e-8226-41a4-870d-597936d9a450"). InnerVolumeSpecName "kube-api-access-4rv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:02.496974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496930    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rv5j\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-kube-api-access-4rv5j\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.496974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496959    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.496974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496969    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0727d83e-8226-41a4-870d-597936d9a450-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.496974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496977    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.496974 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496985    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0727d83e-8226-41a4-870d-597936d9a450-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.497316 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.496994    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0727d83e-8226-41a4-870d-597936d9a450-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:02.602836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.602734    2566 generic.go:356] "Generic (PLEG): container finished" podID="0727d83e-8226-41a4-870d-597936d9a450" containerID="a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23" exitCode=0
Jul 17 15:53:02.602836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.602823    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"
Jul 17 15:53:02.602836 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.602819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerDied","Data":"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"}
Jul 17 15:53:02.603172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.602915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv" event={"ID":"0727d83e-8226-41a4-870d-597936d9a450","Type":"ContainerDied","Data":"99b25973a3eaeb02df19447dafca45242cf055fdca7a0eb536c3f8fbaec8ddf6"}
Jul 17 15:53:02.603172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.602948    2566 scope.go:117] "RemoveContainer" containerID="a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"
Jul 17 15:53:02.604530 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.604502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" event={"ID":"7c8abf17-41e6-4e0d-bc59-43e37d0b86de","Type":"ContainerStarted","Data":"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"}
Jul 17 15:53:02.604669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.604633    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:02.612410 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.612389    2566 scope.go:117] "RemoveContainer" containerID="e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"
Jul 17 15:53:02.620469 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.620450    2566 scope.go:117] "RemoveContainer" containerID="a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"
Jul 17 15:53:02.620764 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:02.620744    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23\": container with ID starting with a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23 not found: ID does not exist" containerID="a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"
Jul 17 15:53:02.620815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.620774    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23"} err="failed to get container status \"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23\": rpc error: code = NotFound desc = could not find container \"a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23\": container with ID starting with a20eacb0df4b44e6737b6056789c7a73fab6eb8dcb71eb386946cca9d98ebe23 not found: ID does not exist"
Jul 17 15:53:02.620815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.620794    2566 scope.go:117] "RemoveContainer" containerID="e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"
Jul 17 15:53:02.621027 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:02.621010    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6\": container with ID starting with e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6 not found: ID does not exist" containerID="e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"
Jul 17 15:53:02.621101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.621032    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6"} err="failed to get container status \"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6\": rpc error: code = NotFound desc = could not find container \"e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6\": container with ID starting with e81d3de01f257b17fa15a96edf6045b3f5c249520d0831dbbc4574c4776688f6 not found: ID does not exist"
Jul 17 15:53:02.625312 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.625260    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" podStartSLOduration=1.936013953 podStartE2EDuration="2.625242684s" totalImagesPullingTime="689.228731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:53:02.623553754 +0000 UTC m=+2632.898702074" watchObservedRunningTime="2026-07-17 15:53:02.625242684 +0000 UTC m=+2632.900391006"
Jul 17 15:53:02.638147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.638116    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:53:02.643017 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:02.642990    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1e887c-openshift-default-5786746d8c-nrtgv"]
Jul 17 15:53:04.241210 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:04.241179    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0727d83e-8226-41a4-870d-597936d9a450" path="/var/lib/kubelet/pods/0727d83e-8226-41a4-870d-597936d9a450/volumes"
Jul 17 15:53:05.744108 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.744041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:53:05.744680 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.744658    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:05.744732 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.744685    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:05.744808 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.744795    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:05.744846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.744814    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:05.749591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.749568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.751640 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.751605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f159e6-openshift-default-dockercfg-phlsf\""
Jul 17 15:53:05.760574 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.760544    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:53:05.830021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.829987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830278 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830278 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830278 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830278 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830442 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnz9\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830442 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.830442 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.830346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.930978 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.930941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.930992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnz9\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931163 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931487 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931487 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931487 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931650 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931760 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931794 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.931853 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.931834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.934223 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.934196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.934413 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.934394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.942005 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.941982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:05.942348 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:05.942332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnz9\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9\") pod \"e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") " pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:06.061383 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.061281    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:06.219403 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.219379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:53:06.222272 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:53:06.222245    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda455e17_2eb9_4955_90a5_419e74c9cf21.slice/crio-b9f1651e86ae79d9a809d83fada7ab05e9938c0c342a63ef0be38fc3b1001ffb WatchSource:0}: Error finding container b9f1651e86ae79d9a809d83fada7ab05e9938c0c342a63ef0be38fc3b1001ffb: Status 404 returned error can't find the container with id b9f1651e86ae79d9a809d83fada7ab05e9938c0c342a63ef0be38fc3b1001ffb
Jul 17 15:53:06.224588 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.224556    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:06.224653 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.224624    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:06.224697 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.224656    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:06.627558 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.627518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerStarted","Data":"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"}
Jul 17 15:53:06.627769 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.627579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerStarted","Data":"b9f1651e86ae79d9a809d83fada7ab05e9938c0c342a63ef0be38fc3b1001ffb"}
Jul 17 15:53:06.660911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:06.660847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podStartSLOduration=1.660829377 podStartE2EDuration="1.660829377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:53:06.658404301 +0000 UTC m=+2636.933552634" watchObservedRunningTime="2026-07-17 15:53:06.660829377 +0000 UTC m=+2636.935977688"
Jul 17 15:53:07.061555 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:07.061515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:07.066640 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:07.066607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:07.632796 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:07.632763    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:07.633660 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:07.633642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:08.621708 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.621635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:08.733465 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.733429    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:53:08.733844 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.733826    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:08.733917 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.733845    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727d83e-8226-41a4-870d-597936d9a450" containerName="istio-proxy"
Jul 17 15:53:08.737138 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.737114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.738991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.738965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f159e6-serving-cert\""
Jul 17 15:53:08.749634 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.749603    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:53:08.759424 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.759388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcr44\" (UniqueName: \"kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.759600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.759445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.759600 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.759538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.860480 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.860449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcr44\" (UniqueName: \"kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.860655 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.860492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.860655 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.860513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.860655 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:08.860592    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-serving-cert: secret "maas-api-e2e-api-b-f159e6-serving-cert" not found
Jul 17 15:53:08.860795 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:08.860659    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls podName:2c530e56-cbff-401a-9fde-e813cab27f28 nodeName:}" failed. No retries permitted until 2026-07-17 15:53:09.360644544 +0000 UTC m=+2639.635792844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls") pod "maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" (UID: "2c530e56-cbff-401a-9fde-e813cab27f28") : secret "maas-api-e2e-api-b-f159e6-serving-cert" not found
Jul 17 15:53:08.861170 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.861152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:08.875495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:08.875420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcr44\" (UniqueName: \"kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:09.366434 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:09.366395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:09.368984 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:09.368961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") pod \"maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:09.650318 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:09.650227    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:09.823570 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:09.823373    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:53:09.826514 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:53:09.826482    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c530e56_cbff_401a_9fde_e813cab27f28.slice/crio-6803d4460a0a6eabc60ce178ff70d4c2d7bd6d4b216375cd20e0992381c5b2aa WatchSource:0}: Error finding container 6803d4460a0a6eabc60ce178ff70d4c2d7bd6d4b216375cd20e0992381c5b2aa: Status 404 returned error can't find the container with id 6803d4460a0a6eabc60ce178ff70d4c2d7bd6d4b216375cd20e0992381c5b2aa
Jul 17 15:53:10.650190 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:10.650158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" event={"ID":"2c530e56-cbff-401a-9fde-e813cab27f28","Type":"ContainerStarted","Data":"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"}
Jul 17 15:53:10.650377 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:10.650213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" event={"ID":"2c530e56-cbff-401a-9fde-e813cab27f28","Type":"ContainerStarted","Data":"6803d4460a0a6eabc60ce178ff70d4c2d7bd6d4b216375cd20e0992381c5b2aa"}
Jul 17 15:53:10.650377 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:10.650236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:10.669454 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:10.669392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" podStartSLOduration=2.319072087 podStartE2EDuration="2.669374937s" totalImagesPullingTime="350.30285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:53:10.667498678 +0000 UTC m=+2640.942646998" watchObservedRunningTime="2026-07-17 15:53:10.669374937 +0000 UTC m=+2640.944523258"
Jul 17 15:53:13.667848 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.667815    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerID="f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e" exitCode=0
Jul 17 15:53:13.667848 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.667853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerDied","Data":"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"}
Jul 17 15:53:13.668385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.668323    2566 scope.go:117] "RemoveContainer" containerID="f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"
Jul 17 15:53:13.669430 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.669400    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:13.669495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.669471    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:13.669532 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:13.669505    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:14.582670 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:14.582629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:14.673951 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:14.673911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerStarted","Data":"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"}
Jul 17 15:53:15.344996 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:15.344953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:15.346295 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:15.346269    2566 patch_prober.go:28] interesting pod/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 15:53:15.346436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:15.346318    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 15:53:16.344782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:16.344746    2566 patch_prober.go:28] interesting pod/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 15:53:16.345223 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:16.344803    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 15:53:16.663091 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:16.662985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:17.344451 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:17.344402    2566 patch_prober.go:28] interesting pod/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 15:53:17.344662 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:17.344483    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 15:53:18.344911 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:18.344863    2566 patch_prober.go:28] interesting pod/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 17 15:53:18.345495 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:18.344919    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 17 15:53:19.348641 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:19.348606    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:19.349118 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:19.348887    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:19.349673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:19.349650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:23.526363 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.526290    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:49120->10.134.0.52:15021: read: connection reset by peer" start-of-body=
Jul 17 15:53:23.526744 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.526393    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:49120->10.134.0.52:15021: read: connection reset by peer"
Jul 17 15:53:23.729514 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.729431    2566 generic.go:356] "Generic (PLEG): container finished" podID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerID="c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785" exitCode=0
Jul 17 15:53:23.729514 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.729470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerDied","Data":"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"}
Jul 17 15:53:23.729886 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.729871    2566 scope.go:117] "RemoveContainer" containerID="c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"
Jul 17 15:53:23.737099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.731617    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:23.737099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.731777    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:23.737099 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:23.731877    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:53:24.736646 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:24.736612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerStarted","Data":"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"}
Jul 17 15:53:25.061861 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:25.061829    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:25.062171 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:25.062138    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 17 15:53:25.062231 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:25.062206    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 17 15:53:26.062591 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:26.062550    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 17 15:53:26.063009 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:26.062618    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 17 15:53:27.062219 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:27.062177    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 17 15:53:27.062408 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:27.062266    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 17 15:53:28.062038 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:28.061999    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Jul 17 15:53:28.062474 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:28.062093    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Jul 17 15:53:29.065906 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:29.065876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:29.066324 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:29.066168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:29.066950 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:29.066931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:53:31.165175 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.165135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:31.165619 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.165413    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" podUID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" containerName="maas-api" containerID="cri-o://bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198" gracePeriod=30
Jul 17 15:53:31.426458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.426399    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:31.594630 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.594591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls\") pod \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") "
Jul 17 15:53:31.594811 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.594645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkcpw\" (UniqueName: \"kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw\") pod \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") "
Jul 17 15:53:31.594811 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.594700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca\") pod \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\" (UID: \"7c8abf17-41e6-4e0d-bc59-43e37d0b86de\") "
Jul 17 15:53:31.595147 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.595123    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c8abf17-41e6-4e0d-bc59-43e37d0b86de" (UID: "7c8abf17-41e6-4e0d-bc59-43e37d0b86de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:53:31.596886 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.596865    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c8abf17-41e6-4e0d-bc59-43e37d0b86de" (UID: "7c8abf17-41e6-4e0d-bc59-43e37d0b86de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:53:31.596937 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.596904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw" (OuterVolumeSpecName: "kube-api-access-lkcpw") pod "7c8abf17-41e6-4e0d-bc59-43e37d0b86de" (UID: "7c8abf17-41e6-4e0d-bc59-43e37d0b86de"). InnerVolumeSpecName "kube-api-access-lkcpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:31.695772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.695678    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:31.695772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.695708    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcpw\" (UniqueName: \"kubernetes.io/projected/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-kube-api-access-lkcpw\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:31.695772 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.695720    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8abf17-41e6-4e0d-bc59-43e37d0b86de-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:31.771883 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.771852    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" containerID="bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198" exitCode=0
Jul 17 15:53:31.772101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.771893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"
Jul 17 15:53:31.772101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.771899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" event={"ID":"7c8abf17-41e6-4e0d-bc59-43e37d0b86de","Type":"ContainerDied","Data":"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"}
Jul 17 15:53:31.772101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.771943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm" event={"ID":"7c8abf17-41e6-4e0d-bc59-43e37d0b86de","Type":"ContainerDied","Data":"5e4475c3cf05661e25a0b6e7d8ae393191b14556863fa23198e6866c6afba984"}
Jul 17 15:53:31.772101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.771965    2566 scope.go:117] "RemoveContainer" containerID="bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"
Jul 17 15:53:31.781251 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.781234    2566 scope.go:117] "RemoveContainer" containerID="bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"
Jul 17 15:53:31.781546 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:31.781526    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198\": container with ID starting with bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198 not found: ID does not exist" containerID="bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"
Jul 17 15:53:31.781593 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.781553    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198"} err="failed to get container status \"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198\": rpc error: code = NotFound desc = could not find container \"bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198\": container with ID starting with bf4f50350c67e9a68f7db1ad47042e49c9f1b7ad46797af5865113b339551198 not found: ID does not exist"
Jul 17 15:53:31.795349 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.795317    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:31.800308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:31.800282    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f54d84-5589f5687c-k4qzm"]
Jul 17 15:53:32.242193 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:32.242160    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" path="/var/lib/kubelet/pods/7c8abf17-41e6-4e0d-bc59-43e37d0b86de/volumes"
Jul 17 15:53:46.757588 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:46.757554    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:53:46.758227 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:46.757778    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" containerID="cri-o://2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc" gracePeriod=30
Jul 17 15:53:49.351292 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:49.351249    2566 patch_prober.go:28] interesting pod/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:53:49.351775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:49.351321    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:53:52.033865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.033795    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:52.078968 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.078936    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.078989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079082    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079103    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079187 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdzfv\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv\") pod \"fe1ba4e9-1937-4859-af54-affcb78d426f\" (UID: \"fe1ba4e9-1937-4859-af54-affcb78d426f\") "
Jul 17 15:53:52.079594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:52.079594 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:53:52.079763 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079615    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1ba4e9-1937-4859-af54-affcb78d426f-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.079763 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079634    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.079763 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:52.080012 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.079837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:52.080324 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.080298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:52.081596 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.081566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:53:52.081719 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.081613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:52.082008 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.081979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv" (OuterVolumeSpecName: "kube-api-access-rdzfv") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "kube-api-access-rdzfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:52.082117 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.082044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe1ba4e9-1937-4859-af54-affcb78d426f" (UID: "fe1ba4e9-1937-4859-af54-affcb78d426f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:53:52.180781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180744    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.180781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180774    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzfv\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-kube-api-access-rdzfv\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.180781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180784    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.180781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180793    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.181051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180800    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.181051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180808    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1ba4e9-1937-4859-af54-affcb78d426f-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.181051 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.180816    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1ba4e9-1937-4859-af54-affcb78d426f-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:52.871368 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.871337    2566 generic.go:356] "Generic (PLEG): container finished" podID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerID="2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc" exitCode=0
Jul 17 15:53:52.871569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.871383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerDied","Data":"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"}
Jul 17 15:53:52.871569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.871390    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"
Jul 17 15:53:52.871569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.871433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj" event={"ID":"fe1ba4e9-1937-4859-af54-affcb78d426f","Type":"ContainerDied","Data":"3401fd146de5b335e2ba189cbaf3baa890d7f4c30c7506cd1711aa91b3ce370f"}
Jul 17 15:53:52.871569 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.871455    2566 scope.go:117] "RemoveContainer" containerID="2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"
Jul 17 15:53:52.880248 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.880218    2566 scope.go:117] "RemoveContainer" containerID="f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"
Jul 17 15:53:52.888407 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.888379    2566 scope.go:117] "RemoveContainer" containerID="2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"
Jul 17 15:53:52.888804 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:52.888781    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc\": container with ID starting with 2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc not found: ID does not exist" containerID="2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"
Jul 17 15:53:52.888886 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.888815    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc"} err="failed to get container status \"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc\": rpc error: code = NotFound desc = could not find container \"2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc\": container with ID starting with 2f4ab6ed9ee3b756c14e599251e7e7b5b390452cd625ba65a1d2c149350917bc not found: ID does not exist"
Jul 17 15:53:52.888886 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.888834    2566 scope.go:117] "RemoveContainer" containerID="f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"
Jul 17 15:53:52.889164 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:52.889139    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e\": container with ID starting with f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e not found: ID does not exist" containerID="f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"
Jul 17 15:53:52.889266 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.889176    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e"} err="failed to get container status \"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e\": rpc error: code = NotFound desc = could not find container \"f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e\": container with ID starting with f1b510fe1ce0c65b590d2c626cf8cc7c619696899deca54debf004f970e1505e not found: ID does not exist"
Jul 17 15:53:52.892307 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.892285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:53:52.896385 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:52.896361    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f54d84-openshift-default-5b868744bf-lvjvj"]
Jul 17 15:53:54.240846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:54.240808    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" path="/var/lib/kubelet/pods/fe1ba4e9-1937-4859-af54-affcb78d426f/volumes"
Jul 17 15:53:57.792812 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:57.792776    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:53:57.793370 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:57.792975    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" podUID="2c530e56-cbff-401a-9fde-e813cab27f28" containerName="maas-api" containerID="cri-o://e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d" gracePeriod=30
Jul 17 15:53:58.045817 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.045748    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:58.139469 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.139427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca\") pod \"2c530e56-cbff-401a-9fde-e813cab27f28\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") "
Jul 17 15:53:58.139664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.139505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcr44\" (UniqueName: \"kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44\") pod \"2c530e56-cbff-401a-9fde-e813cab27f28\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") "
Jul 17 15:53:58.139664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.139557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") pod \"2c530e56-cbff-401a-9fde-e813cab27f28\" (UID: \"2c530e56-cbff-401a-9fde-e813cab27f28\") "
Jul 17 15:53:58.139847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.139820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c530e56-cbff-401a-9fde-e813cab27f28" (UID: "2c530e56-cbff-401a-9fde-e813cab27f28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:53:58.141781 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.141749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c530e56-cbff-401a-9fde-e813cab27f28" (UID: "2c530e56-cbff-401a-9fde-e813cab27f28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:53:58.141879 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.141831    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44" (OuterVolumeSpecName: "kube-api-access-vcr44") pod "2c530e56-cbff-401a-9fde-e813cab27f28" (UID: "2c530e56-cbff-401a-9fde-e813cab27f28"). InnerVolumeSpecName "kube-api-access-vcr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:53:58.241164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.241131    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcr44\" (UniqueName: \"kubernetes.io/projected/2c530e56-cbff-401a-9fde-e813cab27f28-kube-api-access-vcr44\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:58.241164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.241156    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c530e56-cbff-401a-9fde-e813cab27f28-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:58.241164 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.241166    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c530e56-cbff-401a-9fde-e813cab27f28-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:53:58.901268 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.901233    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c530e56-cbff-401a-9fde-e813cab27f28" containerID="e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d" exitCode=0
Jul 17 15:53:58.901707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.901284    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"
Jul 17 15:53:58.901707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.901321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" event={"ID":"2c530e56-cbff-401a-9fde-e813cab27f28","Type":"ContainerDied","Data":"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"}
Jul 17 15:53:58.901707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.901374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w" event={"ID":"2c530e56-cbff-401a-9fde-e813cab27f28","Type":"ContainerDied","Data":"6803d4460a0a6eabc60ce178ff70d4c2d7bd6d4b216375cd20e0992381c5b2aa"}
Jul 17 15:53:58.901707 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.901395    2566 scope.go:117] "RemoveContainer" containerID="e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"
Jul 17 15:53:58.910281 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.910262    2566 scope.go:117] "RemoveContainer" containerID="e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"
Jul 17 15:53:58.910556 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:53:58.910535    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d\": container with ID starting with e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d not found: ID does not exist" containerID="e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"
Jul 17 15:53:58.910622 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.910565    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d"} err="failed to get container status \"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d\": rpc error: code = NotFound desc = could not find container \"e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d\": container with ID starting with e5d3e38e46e4341c194cc98ec15f5e00a4e19e994a0f1afdcfe36d0d2e9a573d not found: ID does not exist"
Jul 17 15:53:58.921679 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.921652    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:53:58.926664 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:53:58.926639    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f159e6-5bb7d98cc5-mhh6w"]
Jul 17 15:54:00.240820 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:00.240739    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c530e56-cbff-401a-9fde-e813cab27f28" path="/var/lib/kubelet/pods/2c530e56-cbff-401a-9fde-e813cab27f28/volumes"
Jul 17 15:54:10.346893 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:10.346863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:54:10.354299 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:10.354276    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:54:12.954963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:12.954922    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:54:12.955450 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:12.955191    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" containerID="cri-o://e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac" gracePeriod=30
Jul 17 15:54:13.965829 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.965793    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966216    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c530e56-cbff-401a-9fde-e813cab27f28" containerName="maas-api"
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966228    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c530e56-cbff-401a-9fde-e813cab27f28" containerName="maas-api"
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966239    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" containerName="maas-api"
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966245    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" containerName="maas-api"
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966275    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966281    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966344    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966353    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1ba4e9-1937-4859-af54-affcb78d426f" containerName="istio-proxy"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966360    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c530e56-cbff-401a-9fde-e813cab27f28" containerName="maas-api"
Jul 17 15:54:13.966460 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.966368    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8abf17-41e6-4e0d-bc59-43e37d0b86de" containerName="maas-api"
Jul 17 15:54:13.970414 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.970394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:13.976018 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.975991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-291633-openshift-default-dockercfg-qs8br\""
Jul 17 15:54:13.988206 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:13.988174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 15:54:14.069262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.069217    2566 patch_prober.go:28] interesting pod/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 15:54:14.069433 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.069290    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 15:54:14.093342 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l956f\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093510 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093492    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093649 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.093948 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.093649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.194780 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.194677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.194780 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.194726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.194780 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.194745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195128 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.194866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195128 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.194937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l956f\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195128 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195128 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195373 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195557 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.195846 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.195825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.197578 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.197559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.197815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.197794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.212402 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.212367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.212542 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.212434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l956f\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f\") pod \"e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") " pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.280899 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.280859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:14.443695 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.443669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 15:54:14.445315 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:54:14.445244    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c97511a_0849_4fa5_8b54_7904edfcb02e.slice/crio-3379f33942a78fcff6bf9ce25bb72f5651e5a2524ba785ad3dd2966ba004fc38 WatchSource:0}: Error finding container 3379f33942a78fcff6bf9ce25bb72f5651e5a2524ba785ad3dd2966ba004fc38: Status 404 returned error can't find the container with id 3379f33942a78fcff6bf9ce25bb72f5651e5a2524ba785ad3dd2966ba004fc38
Jul 17 15:54:14.447334 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.447303    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:14.447412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.447370    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:14.447412 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.447406    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:14.977240 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.977207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerStarted","Data":"b451047504bb0c6072b91b0a96d36b339fec3fee29a08fdb538ee60a87730b04"}
Jul 17 15:54:14.977653 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:14.977262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerStarted","Data":"3379f33942a78fcff6bf9ce25bb72f5651e5a2524ba785ad3dd2966ba004fc38"}
Jul 17 15:54:15.016036 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:15.015986    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podStartSLOduration=2.015971114 podStartE2EDuration="2.015971114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:54:15.014631949 +0000 UTC m=+2705.289780288" watchObservedRunningTime="2026-07-17 15:54:15.015971114 +0000 UTC m=+2705.291119435"
Jul 17 15:54:15.281135 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:15.281047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:15.286184 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:15.286156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:15.981755 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:15.981717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:15.982652 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:15.982635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:17.243643 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.243611    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 15:54:17.246949 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.246924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.250422 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.250394    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-291633-serving-cert\""
Jul 17 15:54:17.250564 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.250512    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vq8kc\""
Jul 17 15:54:17.264110 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.264081    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 15:54:17.328428 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.328382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.328653 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.328554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5kg\" (UniqueName: \"kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.328786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.328660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.429390 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.429346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.429610 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.429418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5kg\" (UniqueName: \"kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.429610 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.429460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.429610 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:17.429518    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-serving-cert: secret "maas-api-e2e-shared-a-291633-serving-cert" not found
Jul 17 15:54:17.429610 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:17.429603    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls podName:0a6ad586-d766-46fa-bde2-25e68c67f7e4 nodeName:}" failed. No retries permitted until 2026-07-17 15:54:17.92958243 +0000 UTC m=+2708.204730732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls") pod "maas-api-e2e-shared-a-291633-f74669977-dr9tp" (UID: "0a6ad586-d766-46fa-bde2-25e68c67f7e4") : secret "maas-api-e2e-shared-a-291633-serving-cert" not found
Jul 17 15:54:17.430157 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.430140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.443617 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.443588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5kg\" (UniqueName: \"kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.934521 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.934479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:17.937209 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:17.937182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-291633-f74669977-dr9tp\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:18.158483 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.158453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:18.204137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.204109    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:54:18.321077 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.321000    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 15:54:18.323714 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:54:18.323687    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6ad586_d766_46fa_bde2_25e68c67f7e4.slice/crio-fcf0a640a9e8f24366503bb60e5a7ef8c3efd2b6a237578153f8fadbe4b61b28 WatchSource:0}: Error finding container fcf0a640a9e8f24366503bb60e5a7ef8c3efd2b6a237578153f8fadbe4b61b28: Status 404 returned error can't find the container with id fcf0a640a9e8f24366503bb60e5a7ef8c3efd2b6a237578153f8fadbe4b61b28
Jul 17 15:54:18.337994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.337956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.337994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.337998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsnz9\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338271 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338271 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338182    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338271 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338430 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338430 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338430 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338430 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs\") pod \"da455e17-2eb9-4955-90a5-419e74c9cf21\" (UID: \"da455e17-2eb9-4955-90a5-419e74c9cf21\") "
Jul 17 15:54:18.338715 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 15:54:18.338863 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338814    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:54:18.338863 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338846    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da455e17-2eb9-4955-90a5-419e74c9cf21-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.339124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.338983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:54:18.339124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.339028    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:54:18.339124 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.339040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data" (OuterVolumeSpecName: "istio-data") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:54:18.341052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.341029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:54:18.341052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.341040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token" (OuterVolumeSpecName: "istio-token") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:54:18.341216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.341121    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9" (OuterVolumeSpecName: "kube-api-access-dsnz9") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "kube-api-access-dsnz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:54:18.341216 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.341168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da455e17-2eb9-4955-90a5-419e74c9cf21" (UID: "da455e17-2eb9-4955-90a5-419e74c9cf21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 15:54:18.439847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439820    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.439847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439844    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.439847 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439853    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.440100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439861    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.440100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439872    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.440100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439880    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.440100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439890    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da455e17-2eb9-4955-90a5-419e74c9cf21-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:18.440100 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:18.439898    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsnz9\" (UniqueName: \"kubernetes.io/projected/da455e17-2eb9-4955-90a5-419e74c9cf21-kube-api-access-dsnz9\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:54:19.000333 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.000303    2566 generic.go:356] "Generic (PLEG): container finished" podID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerID="e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac" exitCode=0
Jul 17 15:54:19.000533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.000355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"
Jul 17 15:54:19.000533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.000396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerDied","Data":"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"}
Jul 17 15:54:19.000533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.000474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw" event={"ID":"da455e17-2eb9-4955-90a5-419e74c9cf21","Type":"ContainerDied","Data":"b9f1651e86ae79d9a809d83fada7ab05e9938c0c342a63ef0be38fc3b1001ffb"}
Jul 17 15:54:19.000533 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.000505    2566 scope.go:117] "RemoveContainer" containerID="e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"
Jul 17 15:54:19.002213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.002185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" event={"ID":"0a6ad586-d766-46fa-bde2-25e68c67f7e4","Type":"ContainerStarted","Data":"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"}
Jul 17 15:54:19.002351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.002230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" event={"ID":"0a6ad586-d766-46fa-bde2-25e68c67f7e4","Type":"ContainerStarted","Data":"fcf0a640a9e8f24366503bb60e5a7ef8c3efd2b6a237578153f8fadbe4b61b28"}
Jul 17 15:54:19.002351 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.002251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:19.009838 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.009822    2566 scope.go:117] "RemoveContainer" containerID="c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"
Jul 17 15:54:19.018518 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.018496    2566 scope.go:117] "RemoveContainer" containerID="e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"
Jul 17 15:54:19.018814 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:19.018792    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac\": container with ID starting with e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac not found: ID does not exist" containerID="e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"
Jul 17 15:54:19.018892 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.018822    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac"} err="failed to get container status \"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac\": rpc error: code = NotFound desc = could not find container \"e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac\": container with ID starting with e0625afe3eed4c406372850661ad82ae81d72ab287d8de8afe38551711bb49ac not found: ID does not exist"
Jul 17 15:54:19.018892 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.018841    2566 scope.go:117] "RemoveContainer" containerID="c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"
Jul 17 15:54:19.019101 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:19.019080    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785\": container with ID starting with c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785 not found: ID does not exist" containerID="c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"
Jul 17 15:54:19.019146 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.019110    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785"} err="failed to get container status \"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785\": rpc error: code = NotFound desc = could not find container \"c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785\": container with ID starting with c02e977818664bd829ec11ad73b73d375397727f89d9349a8731bb256d779785 not found: ID does not exist"
Jul 17 15:54:19.053743 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.053705    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:54:19.065504 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:19.065471    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f159e6-openshift-default-65d99d7585-kt8xw"]
Jul 17 15:54:20.241029 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:20.240991    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" path="/var/lib/kubelet/pods/da455e17-2eb9-4955-90a5-419e74c9cf21/volumes"
Jul 17 15:54:22.256750 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.256698    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" podStartSLOduration=4.863050732 podStartE2EDuration="5.256681331s" totalImagesPullingTime="393.630599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:54:19.118737184 +0000 UTC m=+2709.393885505" watchObservedRunningTime="2026-07-17 15:54:22.256681331 +0000 UTC m=+2712.531829652"
Jul 17 15:54:22.258355 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258332    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 15:54:22.258699 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258688    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:22.258741 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258701    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:22.258741 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258720    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:22.258741 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258725    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:22.258839 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.258833    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:22.263337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.263314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.265453 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.265436    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f65ca6-openshift-default-dockercfg-c25fl\""
Jul 17 15:54:22.278230 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.278205    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 15:54:22.378199 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p7k\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378361 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.378497 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.378457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479635 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p7k\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.479857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.479855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.480286 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.480099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.480286 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.480151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.480479 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.480457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.480543 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.480495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.480598 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.480582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.482242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.482211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.482605 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.482581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.488994 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.488967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.489213 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.489197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p7k\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k\") pod \"e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") " pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.573311 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.573219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:22.719673 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.719645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 15:54:22.722174 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:54:22.722143    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55d1709_e81a_4273_855b_cd685ca6dab8.slice/crio-96ab678cf91c520a8431413817b6903abd7bf51780fb1b7273e38b6f42282535 WatchSource:0}: Error finding container 96ab678cf91c520a8431413817b6903abd7bf51780fb1b7273e38b6f42282535: Status 404 returned error can't find the container with id 96ab678cf91c520a8431413817b6903abd7bf51780fb1b7273e38b6f42282535
Jul 17 15:54:22.724528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.724493    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:22.724608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.724563    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:22.724608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:22.724594    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:23.025963 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:23.025914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerStarted","Data":"9985fee20a8fa92c2a58bf946db3109e014d582215aae582cb21d78d0b9cdb50"}
Jul 17 15:54:23.026178 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:23.025985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerStarted","Data":"96ab678cf91c520a8431413817b6903abd7bf51780fb1b7273e38b6f42282535"}
Jul 17 15:54:23.051044 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:23.050987    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podStartSLOduration=1.050972557 podStartE2EDuration="1.050972557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:54:23.048678112 +0000 UTC m=+2713.323826435" watchObservedRunningTime="2026-07-17 15:54:23.050972557 +0000 UTC m=+2713.326120878"
Jul 17 15:54:23.573575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:23.573533    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:23.578455 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:23.578427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:24.031244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:24.031211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:24.032263 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:24.032244    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:25.019177 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.019145    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 15:54:25.325427 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.325328    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 15:54:25.325871 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.325851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da455e17-2eb9-4955-90a5-419e74c9cf21" containerName="istio-proxy"
Jul 17 15:54:25.329026 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.329001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.333587 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.333559    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-f65ca6-serving-cert\""
Jul 17 15:54:25.351782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.351660    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 15:54:25.423452 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.423399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.423452 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.423449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz6pk\" (UniqueName: \"kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.423669 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.423630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.524754 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.524709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.524940 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.524765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.524940 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.524787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz6pk\" (UniqueName: \"kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.524940 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:25.524867    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-serving-cert: secret "maas-api-e2e-shared-b-f65ca6-serving-cert" not found
Jul 17 15:54:25.525133 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:54:25.524944    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls podName:59fd33d9-b39b-452c-ab22-dd881fc613cf nodeName:}" failed. No retries permitted until 2026-07-17 15:54:26.02492349 +0000 UTC m=+2716.300071792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls") pod "maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" (UID: "59fd33d9-b39b-452c-ab22-dd881fc613cf") : secret "maas-api-e2e-shared-b-f65ca6-serving-cert" not found
Jul 17 15:54:25.525386 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.525366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:25.538384 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:25.538357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz6pk\" (UniqueName: \"kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:26.029955 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:26.029912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:26.032608 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:26.032584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:26.240280 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:26.240232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:26.442718 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:26.442686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 15:54:26.446445 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:54:26.446414    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fd33d9_b39b_452c_ab22_dd881fc613cf.slice/crio-d07a85a762e97a8e26c9beec973acc2d70382c5addae4aaef32c52794b9055f3 WatchSource:0}: Error finding container d07a85a762e97a8e26c9beec973acc2d70382c5addae4aaef32c52794b9055f3: Status 404 returned error can't find the container with id d07a85a762e97a8e26c9beec973acc2d70382c5addae4aaef32c52794b9055f3
Jul 17 15:54:27.046381 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:27.046338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" event={"ID":"59fd33d9-b39b-452c-ab22-dd881fc613cf","Type":"ContainerStarted","Data":"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"}
Jul 17 15:54:27.046777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:27.046446    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:27.046777 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:27.046471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" event={"ID":"59fd33d9-b39b-452c-ab22-dd881fc613cf","Type":"ContainerStarted","Data":"d07a85a762e97a8e26c9beec973acc2d70382c5addae4aaef32c52794b9055f3"}
Jul 17 15:54:27.067299 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:27.067197    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" podStartSLOduration=1.716487171 podStartE2EDuration="2.067179686s" totalImagesPullingTime="350.692515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:54:27.0650348 +0000 UTC m=+2717.340183122" watchObservedRunningTime="2026-07-17 15:54:27.067179686 +0000 UTC m=+2717.342328007"
Jul 17 15:54:30.983554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:30.983440    2566 patch_prober.go:28] interesting pod/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 17 15:54:30.983554 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:30.983508    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 17 15:54:31.069098 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.069046    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerID="b451047504bb0c6072b91b0a96d36b339fec3fee29a08fdb538ee60a87730b04" exitCode=0
Jul 17 15:54:31.069330 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.069129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerDied","Data":"b451047504bb0c6072b91b0a96d36b339fec3fee29a08fdb538ee60a87730b04"}
Jul 17 15:54:31.069629 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.069611    2566 scope.go:117] "RemoveContainer" containerID="b451047504bb0c6072b91b0a96d36b339fec3fee29a08fdb538ee60a87730b04"
Jul 17 15:54:31.070713 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.070679    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:31.070858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.070778    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:31.070858 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:31.070828    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:32.076595 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:32.076559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerStarted","Data":"5bec15287f5e56650167a086c1f5a7d05578f01a5d5d3ba1b3b50e59f168ac22"}
Jul 17 15:54:32.282140 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:32.282106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:32.282416 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:32.282328    2566 patch_prober.go:28] interesting pod/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 17 15:54:32.282526 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:32.282444    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 17 15:54:33.058897 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:33.058858    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 15:54:33.282105 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:33.282043    2566 patch_prober.go:28] interesting pod/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 17 15:54:33.282511 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:33.282126    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 17 15:54:34.281815 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:34.281773    2566 patch_prober.go:28] interesting pod/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 17 15:54:34.282011 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:34.281836    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 17 15:54:35.281782 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:35.281741    2566 patch_prober.go:28] interesting pod/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 17 15:54:35.282200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:35.281810    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 17 15:54:36.285501 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:36.285468    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:36.285888 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:36.285819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:36.286737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:36.286716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 15:54:38.109757 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.109716    2566 generic.go:356] "Generic (PLEG): container finished" podID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerID="9985fee20a8fa92c2a58bf946db3109e014d582215aae582cb21d78d0b9cdb50" exitCode=0
Jul 17 15:54:38.110234 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.109792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerDied","Data":"9985fee20a8fa92c2a58bf946db3109e014d582215aae582cb21d78d0b9cdb50"}
Jul 17 15:54:38.110353 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.110334    2566 scope.go:117] "RemoveContainer" containerID="9985fee20a8fa92c2a58bf946db3109e014d582215aae582cb21d78d0b9cdb50"
Jul 17 15:54:38.111402 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.111376    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:38.111469 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.111449    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:38.111507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:38.111486    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 15:54:39.031339 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:39.031289    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:39.120259 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:39.120216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerStarted","Data":"6e865d012989d92f7b0f35ceece6a8b3fe570503dd7c0e017fe7e958b4c81844"}
Jul 17 15:54:39.573783 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:39.573743    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:39.574898 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:39.574879    2566 patch_prober.go:28] interesting pod/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 17 15:54:39.574975 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:39.574921    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 17 15:54:40.573946 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:40.573906    2566 patch_prober.go:28] interesting pod/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 17 15:54:40.574436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:40.573970    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 17 15:54:41.574397 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:41.574364    2566 patch_prober.go:28] interesting pod/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 17 15:54:41.574800 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:41.574421    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 17 15:54:42.574325 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:42.574284    2566 patch_prober.go:28] interesting pod/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 17 15:54:42.574543 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:42.574381    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 17 15:54:43.580185 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:43.580154    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:43.580661 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:43.580626    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:54:43.581458 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:54:43.581441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 15:55:04.221254 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.221215    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:55:04.224935 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.224917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.227983 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.227958    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"kube-root-ca.crt\""
Jul 17 15:55:04.228172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.227959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"auth-test-model-f65ca6-kserve-self-signed-certs\""
Jul 17 15:55:04.228172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.227966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"default-dockercfg-gzqq4\""
Jul 17 15:55:04.228172 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.228050    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"openshift-service-ca.crt\""
Jul 17 15:55:04.248242 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.248210    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:55:04.326675 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzkkk\" (UniqueName: \"kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.326865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.326865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.326865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.326865 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.327126 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.326895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.427921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.427880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzkkk\" (UniqueName: \"kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.427932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.427972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428159 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428507 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428575 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.428613 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.428566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.430667 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.430643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.430890 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.430871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.438939 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.438914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzkkk\" (UniqueName: \"kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk\") pod \"auth-test-model-f65ca6-kserve-85dc47f75-r2hb2\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") " pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.536797 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.536755    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:04.681048 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:04.680991    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:55:04.683406 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:55:04.683370    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b815a7f_01c2_402b_b324_e6f49cafdb0c.slice/crio-9d327e1f02766f51a72544c35db155b8ae0d1f0f0bacde89d10d1457fd61217b WatchSource:0}: Error finding container 9d327e1f02766f51a72544c35db155b8ae0d1f0f0bacde89d10d1457fd61217b: Status 404 returned error can't find the container with id 9d327e1f02766f51a72544c35db155b8ae0d1f0f0bacde89d10d1457fd61217b
Jul 17 15:55:05.252566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:05.252529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" event={"ID":"8b815a7f-01c2-402b-b324-e6f49cafdb0c","Type":"ContainerStarted","Data":"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"}
Jul 17 15:55:05.252566 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:05.252572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" event={"ID":"8b815a7f-01c2-402b-b324-e6f49cafdb0c","Type":"ContainerStarted","Data":"9d327e1f02766f51a72544c35db155b8ae0d1f0f0bacde89d10d1457fd61217b"}
Jul 17 15:55:05.274337 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:05.274281    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" podStartSLOduration=1.274259308 podStartE2EDuration="1.274259308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:55:05.271788233 +0000 UTC m=+2755.546936557" watchObservedRunningTime="2026-07-17 15:55:05.274259308 +0000 UTC m=+2755.549407654"
Jul 17 15:55:14.537775 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:14.537731    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:14.538219 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:14.537792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:14.550199 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:14.550173    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:55:15.312813 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:55:15.312784    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:57:07.113605 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:07.113523    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:57:22.134125 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.134020    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" podUID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" containerName="main" containerID="cri-o://b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30" gracePeriod=45
Jul 17 15:57:22.383282 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.383252    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:57:22.475540 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475540 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzkkk\" (UniqueName: \"kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475650    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475698    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs\") pod \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\" (UID: \"8b815a7f-01c2-402b-b324-e6f49cafdb0c\") "
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:57:22.475786 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.475765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home" (OuterVolumeSpecName: "home") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:57:22.476097 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.476023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:57:22.476097 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.476041    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-home\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.476097 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.476079    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-model-cache\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.478137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.478103    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm" (OuterVolumeSpecName: "dshm") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:57:22.478137 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.478123    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk" (OuterVolumeSpecName: "kube-api-access-wzkkk") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "kube-api-access-wzkkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:57:22.478298 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.478159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b815a7f-01c2-402b-b324-e6f49cafdb0c" (UID: "8b815a7f-01c2-402b-b324-e6f49cafdb0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:57:22.577528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.577491    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzkkk\" (UniqueName: \"kubernetes.io/projected/8b815a7f-01c2-402b-b324-e6f49cafdb0c-kube-api-access-wzkkk\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.577528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.577520    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tmp-dir\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.577528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.577530    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b815a7f-01c2-402b-b324-e6f49cafdb0c-dshm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.577528 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.577539    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b815a7f-01c2-402b-b324-e6f49cafdb0c-tls-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:57:22.934042 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.934003    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" containerID="b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30" exitCode=0
Jul 17 15:57:22.934042 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.934050    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"
Jul 17 15:57:22.934315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.934098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" event={"ID":"8b815a7f-01c2-402b-b324-e6f49cafdb0c","Type":"ContainerDied","Data":"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"}
Jul 17 15:57:22.934315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.934144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2" event={"ID":"8b815a7f-01c2-402b-b324-e6f49cafdb0c","Type":"ContainerDied","Data":"9d327e1f02766f51a72544c35db155b8ae0d1f0f0bacde89d10d1457fd61217b"}
Jul 17 15:57:22.934315 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.934163    2566 scope.go:117] "RemoveContainer" containerID="b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"
Jul 17 15:57:22.943798 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.943778    2566 scope.go:117] "RemoveContainer" containerID="b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"
Jul 17 15:57:22.944138 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:57:22.944118    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30\": container with ID starting with b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30 not found: ID does not exist" containerID="b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"
Jul 17 15:57:22.944200 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.944160    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30"} err="failed to get container status \"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30\": rpc error: code = NotFound desc = could not find container \"b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30\": container with ID starting with b051a3e6d8652ad84785cf052c6958bbd953ad6826cd9759728be67e6def1c30 not found: ID does not exist"
Jul 17 15:57:22.960745 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.960710    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:57:22.963151 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:22.963124    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/auth-test-model-f65ca6-kserve-85dc47f75-r2hb2"]
Jul 17 15:57:24.241610 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:24.241577    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" path="/var/lib/kubelet/pods/8b815a7f-01c2-402b-b324-e6f49cafdb0c/volumes"
Jul 17 15:57:45.319436 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.319398    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:57:45.319900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.319812    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" containerName="main"
Jul 17 15:57:45.319900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.319823    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" containerName="main"
Jul 17 15:57:45.319900 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.319879    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b815a7f-01c2-402b-b324-e6f49cafdb0c" containerName="main"
Jul 17 15:57:45.324070 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.324038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.326382 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.326362    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"openshift-service-ca.crt\""
Jul 17 15:57:45.327074 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.327041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"default-dockercfg-gzqq4\""
Jul 17 15:57:45.327209 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.327041    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"kube-root-ca.crt\""
Jul 17 15:57:45.327308 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.327212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"sub-test-model-f65ca6-kserve-self-signed-certs\""
Jul 17 15:57:45.334898 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.334873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:57:45.403434 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.403434 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.403737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.403737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.403737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctvd\" (UniqueName: \"kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.403737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.403722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505047 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505256 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctvd\" (UniqueName: \"kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505256 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505256 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505256 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505484 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505576 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505550    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505737 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.505857 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.505715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.507629 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.507597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.507788 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.507773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.515753 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.515721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctvd\" (UniqueName: \"kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd\") pod \"sub-test-model-f65ca6-kserve-547df98fc7-ftppg\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") " pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.636246 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.636137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:45.779410 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:45.779377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:57:45.782003 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:57:45.781966    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d03b37_69ff_4a68_ae9a_91c8dcd343f5.slice/crio-889d831576ce54112000109ed8d2ac7f22891a56cd40f6928afdfd1d03deb642 WatchSource:0}: Error finding container 889d831576ce54112000109ed8d2ac7f22891a56cd40f6928afdfd1d03deb642: Status 404 returned error can't find the container with id 889d831576ce54112000109ed8d2ac7f22891a56cd40f6928afdfd1d03deb642
Jul 17 15:57:46.052517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:46.052480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" event={"ID":"88d03b37-69ff-4a68-ae9a-91c8dcd343f5","Type":"ContainerStarted","Data":"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"}
Jul 17 15:57:46.052517 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:46.052521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" event={"ID":"88d03b37-69ff-4a68-ae9a-91c8dcd343f5","Type":"ContainerStarted","Data":"889d831576ce54112000109ed8d2ac7f22891a56cd40f6928afdfd1d03deb642"}
Jul 17 15:57:46.071406 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:46.071345    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" podStartSLOduration=1.071328945 podStartE2EDuration="1.071328945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:57:46.069649036 +0000 UTC m=+2916.344797380" watchObservedRunningTime="2026-07-17 15:57:46.071328945 +0000 UTC m=+2916.346477265"
Jul 17 15:57:55.636282 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:55.636240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:55.636784 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:55.636302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:55.649212 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:55.649180    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:57:56.111523 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:57:56.111483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:58:37.624021 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:37.623322    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:58:52.646534 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.646492    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" podUID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" containerName="main" containerID="cri-o://56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc" gracePeriod=45
Jul 17 15:58:52.901418 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.901340    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:58:52.982394 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982494    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982531    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982626 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982833 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cctvd\" (UniqueName: \"kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd\") pod \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\" (UID: \"88d03b37-69ff-4a68-ae9a-91c8dcd343f5\") "
Jul 17 15:58:52.982833 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982686    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home" (OuterVolumeSpecName: "home") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:58:52.982928 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:58:52.982928 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:58:52.983052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.982997    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tmp-dir\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:52.983052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.983014    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-model-cache\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:52.983052 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.983023    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-home\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:52.984764 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.984740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm" (OuterVolumeSpecName: "dshm") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 15:58:52.984873 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.984854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 15:58:52.984921 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:52.984881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd" (OuterVolumeSpecName: "kube-api-access-cctvd") pod "88d03b37-69ff-4a68-ae9a-91c8dcd343f5" (UID: "88d03b37-69ff-4a68-ae9a-91c8dcd343f5"). InnerVolumeSpecName "kube-api-access-cctvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 15:58:53.084073 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.084031    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-dshm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:53.084244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.084100    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-tls-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:53.084244 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.084119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctvd\" (UniqueName: \"kubernetes.io/projected/88d03b37-69ff-4a68-ae9a-91c8dcd343f5-kube-api-access-cctvd\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 15:58:53.381079 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.381020    2566 generic.go:356] "Generic (PLEG): container finished" podID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" containerID="56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc" exitCode=0
Jul 17 15:58:53.381262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.381084    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"
Jul 17 15:58:53.381262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.381121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" event={"ID":"88d03b37-69ff-4a68-ae9a-91c8dcd343f5","Type":"ContainerDied","Data":"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"}
Jul 17 15:58:53.381262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.381169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg" event={"ID":"88d03b37-69ff-4a68-ae9a-91c8dcd343f5","Type":"ContainerDied","Data":"889d831576ce54112000109ed8d2ac7f22891a56cd40f6928afdfd1d03deb642"}
Jul 17 15:58:53.381262 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.381190    2566 scope.go:117] "RemoveContainer" containerID="56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"
Jul 17 15:58:53.390637 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.390618    2566 scope.go:117] "RemoveContainer" containerID="56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"
Jul 17 15:58:53.390934 ip-10-0-128-207 kubenswrapper[2566]: E0717 15:58:53.390914    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc\": container with ID starting with 56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc not found: ID does not exist" containerID="56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"
Jul 17 15:58:53.390997 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.390948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc"} err="failed to get container status \"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc\": rpc error: code = NotFound desc = could not find container \"56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc\": container with ID starting with 56f84b403a806a0e77f8a74f2b2c0c5bb15c418a705713463c71d5d47dd911cc not found: ID does not exist"
Jul 17 15:58:53.403466 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.403417    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:58:53.408666 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:53.408637    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/sub-test-model-f65ca6-kserve-547df98fc7-ftppg"]
Jul 17 15:58:54.241071 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:58:54.241023    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" path="/var/lib/kubelet/pods/88d03b37-69ff-4a68-ae9a-91c8dcd343f5/volumes"
Jul 17 15:59:10.383025 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:10.382992    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:59:10.391990 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:10.391964    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxh8w_aad909eb-13a9-4390-aab9-ea7f1497d31b/ovn-acl-logging/0.log"
Jul 17 15:59:16.612701 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.612660    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 15:59:16.613267 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.613245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" containerName="main"
Jul 17 15:59:16.613364 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.613270    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" containerName="main"
Jul 17 15:59:16.613419 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.613369    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d03b37-69ff-4a68-ae9a-91c8dcd343f5" containerName="main"
Jul 17 15:59:16.617749 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.617725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.619970 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.619947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"kube-root-ca.crt\""
Jul 17 15:59:16.620756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.620734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"rate-test-model-f65ca6-kserve-self-signed-certs\""
Jul 17 15:59:16.620756 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.620748    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"default-dockercfg-gzqq4\""
Jul 17 15:59:16.620938 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.620862    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f65ca6\"/\"openshift-service-ca.crt\""
Jul 17 15:59:16.624304 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.624278    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 15:59:16.711753 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.711947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.711947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7f9\" (UniqueName: \"kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.711947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.711947 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.712148 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.711979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812478 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812654 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812654 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812654 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7f9\" (UniqueName: \"kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812870 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.812991 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.812966    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.813101 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.813006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.815031 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.815000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.815190 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.815173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.821567 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.821545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7f9\" (UniqueName: \"kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9\") pod \"rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") " pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:16.929778 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:16.929682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:17.077236 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:17.077206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 15:59:17.079927 ip-10-0-128-207 kubenswrapper[2566]: W0717 15:59:17.079894    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7c6b7ff_c111_4990_b24b_24649479e17c.slice/crio-61e404408dbdef098fccae7e84bda0d2d9f8cc16405336f895c1f2b4715b902c WatchSource:0}: Error finding container 61e404408dbdef098fccae7e84bda0d2d9f8cc16405336f895c1f2b4715b902c: Status 404 returned error can't find the container with id 61e404408dbdef098fccae7e84bda0d2d9f8cc16405336f895c1f2b4715b902c
Jul 17 15:59:17.497824 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:17.497784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" event={"ID":"f7c6b7ff-c111-4990-b24b-24649479e17c","Type":"ContainerStarted","Data":"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"}
Jul 17 15:59:17.497824 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:17.497831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" event={"ID":"f7c6b7ff-c111-4990-b24b-24649479e17c","Type":"ContainerStarted","Data":"61e404408dbdef098fccae7e84bda0d2d9f8cc16405336f895c1f2b4715b902c"}
Jul 17 15:59:17.516353 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:17.516293    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" podStartSLOduration=1.516269609 podStartE2EDuration="1.516269609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 15:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 15:59:17.515152132 +0000 UTC m=+3007.790300454" watchObservedRunningTime="2026-07-17 15:59:17.516269609 +0000 UTC m=+3007.791417934"
Jul 17 15:59:26.930725 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:26.930682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:26.930725 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:26.930737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:26.943388 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:26.943365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:27.556457 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:27.556429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 15:59:59.794468 ip-10-0-128-207 kubenswrapper[2566]: I0717 15:59:59.794429    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 16:00:14.816449 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:14.816408    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" podUID="f7c6b7ff-c111-4990-b24b-24649479e17c" containerName="main" containerID="cri-o://67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54" gracePeriod=45
Jul 17 16:00:15.079178 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.079146    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 16:00:15.176048 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176275 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176275 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7f9\" (UniqueName: \"kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176275 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176275 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176275 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache\") pod \"f7c6b7ff-c111-4990-b24b-24649479e17c\" (UID: \"f7c6b7ff-c111-4990-b24b-24649479e17c\") "
Jul 17 16:00:15.176541 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:15.176541 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home" (OuterVolumeSpecName: "home") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:15.176541 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:15.176709 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.176580    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-tmp-dir\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.178538 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.178513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm" (OuterVolumeSpecName: "dshm") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:15.178633 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.178562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:00:15.178633 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.178565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9" (OuterVolumeSpecName: "kube-api-access-nw7f9") pod "f7c6b7ff-c111-4990-b24b-24649479e17c" (UID: "f7c6b7ff-c111-4990-b24b-24649479e17c"). InnerVolumeSpecName "kube-api-access-nw7f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:00:15.277420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.277384    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-home\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.277420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.277415    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-model-cache\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.277420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.277424    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c6b7ff-c111-4990-b24b-24649479e17c-tls-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.277669 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.277433    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7f9\" (UniqueName: \"kubernetes.io/projected/f7c6b7ff-c111-4990-b24b-24649479e17c-kube-api-access-nw7f9\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.277669 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.277442    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c6b7ff-c111-4990-b24b-24649479e17c-dshm\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:15.782383 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.782349    2566 generic.go:356] "Generic (PLEG): container finished" podID="f7c6b7ff-c111-4990-b24b-24649479e17c" containerID="67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54" exitCode=0
Jul 17 16:00:15.782591 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.782399    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"
Jul 17 16:00:15.782591 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.782414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" event={"ID":"f7c6b7ff-c111-4990-b24b-24649479e17c","Type":"ContainerDied","Data":"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"}
Jul 17 16:00:15.782591 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.782444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm" event={"ID":"f7c6b7ff-c111-4990-b24b-24649479e17c","Type":"ContainerDied","Data":"61e404408dbdef098fccae7e84bda0d2d9f8cc16405336f895c1f2b4715b902c"}
Jul 17 16:00:15.782591 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.782461    2566 scope.go:117] "RemoveContainer" containerID="67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"
Jul 17 16:00:15.791846 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.791824    2566 scope.go:117] "RemoveContainer" containerID="67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"
Jul 17 16:00:15.792143 ip-10-0-128-207 kubenswrapper[2566]: E0717 16:00:15.792124    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54\": container with ID starting with 67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54 not found: ID does not exist" containerID="67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"
Jul 17 16:00:15.792204 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.792153    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54"} err="failed to get container status \"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54\": rpc error: code = NotFound desc = could not find container \"67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54\": container with ID starting with 67d2d1c52603cc07f0b39b376e75a0be58905d59b328beb0e5be4916e2f2dd54 not found: ID does not exist"
Jul 17 16:00:15.805134 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.805107    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 16:00:15.810005 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:15.809981    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f65ca6/rate-test-model-f65ca6-kserve-7b7f458cd7-h9wcm"]
Jul 17 16:00:16.241448 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:16.241418    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c6b7ff-c111-4990-b24b-24649479e17c" path="/var/lib/kubelet/pods/f7c6b7ff-c111-4990-b24b-24649479e17c/volumes"
Jul 17 16:00:30.448692 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.448653    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 16:00:30.449177 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.448916    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" podUID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" containerName="maas-api" containerID="cri-o://87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8" gracePeriod=30
Jul 17 16:00:30.712573 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.712504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 16:00:30.728187 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.728151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") pod \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") "
Jul 17 16:00:30.728425 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.728204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca\") pod \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") "
Jul 17 16:00:30.728425 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.728255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd5kg\" (UniqueName: \"kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg\") pod \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\" (UID: \"0a6ad586-d766-46fa-bde2-25e68c67f7e4\") "
Jul 17 16:00:30.728634 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.728532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a6ad586-d766-46fa-bde2-25e68c67f7e4" (UID: "0a6ad586-d766-46fa-bde2-25e68c67f7e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:00:30.731865 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.731830    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg" (OuterVolumeSpecName: "kube-api-access-fd5kg") pod "0a6ad586-d766-46fa-bde2-25e68c67f7e4" (UID: "0a6ad586-d766-46fa-bde2-25e68c67f7e4"). InnerVolumeSpecName "kube-api-access-fd5kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:00:30.732050 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.731874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a6ad586-d766-46fa-bde2-25e68c67f7e4" (UID: "0a6ad586-d766-46fa-bde2-25e68c67f7e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:00:30.829165 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.829129    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a6ad586-d766-46fa-bde2-25e68c67f7e4-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:30.829165 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.829162    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6ad586-d766-46fa-bde2-25e68c67f7e4-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:30.829165 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.829172    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5kg\" (UniqueName: \"kubernetes.io/projected/0a6ad586-d766-46fa-bde2-25e68c67f7e4-kube-api-access-fd5kg\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:30.855351 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.855308    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" containerID="87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8" exitCode=0
Jul 17 16:00:30.855351 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.855353    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"
Jul 17 16:00:30.855606 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.855388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" event={"ID":"0a6ad586-d766-46fa-bde2-25e68c67f7e4","Type":"ContainerDied","Data":"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"}
Jul 17 16:00:30.855606 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.855442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp" event={"ID":"0a6ad586-d766-46fa-bde2-25e68c67f7e4","Type":"ContainerDied","Data":"fcf0a640a9e8f24366503bb60e5a7ef8c3efd2b6a237578153f8fadbe4b61b28"}
Jul 17 16:00:30.855606 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.855467    2566 scope.go:117] "RemoveContainer" containerID="87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"
Jul 17 16:00:30.864354 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.864338    2566 scope.go:117] "RemoveContainer" containerID="87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"
Jul 17 16:00:30.864647 ip-10-0-128-207 kubenswrapper[2566]: E0717 16:00:30.864629    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8\": container with ID starting with 87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8 not found: ID does not exist" containerID="87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"
Jul 17 16:00:30.864686 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.864658    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8"} err="failed to get container status \"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8\": rpc error: code = NotFound desc = could not find container \"87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8\": container with ID starting with 87c4a26cc0a7a66be7ae2ae59ed2053ce90da5e4697182536995560933fc88b8 not found: ID does not exist"
Jul 17 16:00:30.878074 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.878029    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 16:00:30.882821 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:30.882793    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-291633-f74669977-dr9tp"]
Jul 17 16:00:32.241662 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:32.241628    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" path="/var/lib/kubelet/pods/0a6ad586-d766-46fa-bde2-25e68c67f7e4/volumes"
Jul 17 16:00:45.741894 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:45.741859    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 16:00:45.742434 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:45.742095    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy" containerID="cri-o://5bec15287f5e56650167a086c1f5a7d05578f01a5d5d3ba1b3b50e59f168ac22" gracePeriod=30
Jul 17 16:00:50.954448 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:50.954414    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerID="5bec15287f5e56650167a086c1f5a7d05578f01a5d5d3ba1b3b50e59f168ac22" exitCode=0
Jul 17 16:00:50.954865 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:50.954498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerDied","Data":"5bec15287f5e56650167a086c1f5a7d05578f01a5d5d3ba1b3b50e59f168ac22"}
Jul 17 16:00:50.954865 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:50.954576    2566 scope.go:117] "RemoveContainer" containerID="b451047504bb0c6072b91b0a96d36b339fec3fee29a08fdb538ee60a87730b04"
Jul 17 16:00:50.993974 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:50.993941    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 16:00:51.127391 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127391 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l956f\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127592    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127643    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket\") pod \"6c97511a-0849-4fa5-8b54-7904edfcb02e\" (UID: \"6c97511a-0849-4fa5-8b54-7904edfcb02e\") "
Jul 17 16:00:51.127730 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:51.128216 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.127927    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:00:51.128216 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.128016    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.128216 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.128035    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c97511a-0849-4fa5-8b54-7904edfcb02e-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.128216 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.128201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:51.128444 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.128423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:51.128661 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.128636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:51.129791 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.129769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:00:51.129791 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.129782    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 16:00:51.130242 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.130223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:00:51.130242 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.130230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f" (OuterVolumeSpecName: "kube-api-access-l956f") pod "6c97511a-0849-4fa5-8b54-7904edfcb02e" (UID: "6c97511a-0849-4fa5-8b54-7904edfcb02e"). InnerVolumeSpecName "kube-api-access-l956f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:00:51.229136 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229095    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229136 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229126    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229136 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229135    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229136 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229144    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229154    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229163    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l956f\" (UniqueName: \"kubernetes.io/projected/6c97511a-0849-4fa5-8b54-7904edfcb02e-kube-api-access-l956f\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.229420 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.229171    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c97511a-0849-4fa5-8b54-7904edfcb02e-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:51.961603 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.961572    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"
Jul 17 16:00:51.962103 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.961572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph" event={"ID":"6c97511a-0849-4fa5-8b54-7904edfcb02e","Type":"ContainerDied","Data":"3379f33942a78fcff6bf9ce25bb72f5651e5a2524ba785ad3dd2966ba004fc38"}
Jul 17 16:00:51.962103 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.961705    2566 scope.go:117] "RemoveContainer" containerID="5bec15287f5e56650167a086c1f5a7d05578f01a5d5d3ba1b3b50e59f168ac22"
Jul 17 16:00:51.985222 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.985186    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 16:00:51.989591 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:51.989549    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-291633-openshift-default-dc7b9b595-2gsph"]
Jul 17 16:00:52.241495 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:52.241469    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" path="/var/lib/kubelet/pods/6c97511a-0849-4fa5-8b54-7904edfcb02e/volumes"
Jul 17 16:00:56.688134 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.688095    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 16:00:56.688614 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.688342    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" podUID="59fd33d9-b39b-452c-ab22-dd881fc613cf" containerName="maas-api" containerID="cri-o://0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d" gracePeriod=30
Jul 17 16:00:56.942643 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.942572    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 16:00:56.987026 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.986988    2566 generic.go:356] "Generic (PLEG): container finished" podID="59fd33d9-b39b-452c-ab22-dd881fc613cf" containerID="0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d" exitCode=0
Jul 17 16:00:56.987249 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.987035    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"
Jul 17 16:00:56.987249 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.987085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" event={"ID":"59fd33d9-b39b-452c-ab22-dd881fc613cf","Type":"ContainerDied","Data":"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"}
Jul 17 16:00:56.987249 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.987139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8" event={"ID":"59fd33d9-b39b-452c-ab22-dd881fc613cf","Type":"ContainerDied","Data":"d07a85a762e97a8e26c9beec973acc2d70382c5addae4aaef32c52794b9055f3"}
Jul 17 16:00:56.987249 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.987161    2566 scope.go:117] "RemoveContainer" containerID="0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"
Jul 17 16:00:56.995996 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.995972    2566 scope.go:117] "RemoveContainer" containerID="0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"
Jul 17 16:00:56.996327 ip-10-0-128-207 kubenswrapper[2566]: E0717 16:00:56.996306    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d\": container with ID starting with 0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d not found: ID does not exist" containerID="0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"
Jul 17 16:00:56.996380 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:56.996334    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d"} err="failed to get container status \"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d\": rpc error: code = NotFound desc = could not find container \"0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d\": container with ID starting with 0b17f4e2f71b58c20474ef0d45665177ab08607f898d33ff76c64cced431e49d not found: ID does not exist"
Jul 17 16:00:57.086733 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.086701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") pod \"59fd33d9-b39b-452c-ab22-dd881fc613cf\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") "
Jul 17 16:00:57.086885 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.086760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca\") pod \"59fd33d9-b39b-452c-ab22-dd881fc613cf\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") "
Jul 17 16:00:57.086885 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.086818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz6pk\" (UniqueName: \"kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk\") pod \"59fd33d9-b39b-452c-ab22-dd881fc613cf\" (UID: \"59fd33d9-b39b-452c-ab22-dd881fc613cf\") "
Jul 17 16:00:57.087240 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.087207    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59fd33d9-b39b-452c-ab22-dd881fc613cf" (UID: "59fd33d9-b39b-452c-ab22-dd881fc613cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:00:57.089135 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.089110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59fd33d9-b39b-452c-ab22-dd881fc613cf" (UID: "59fd33d9-b39b-452c-ab22-dd881fc613cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 16:00:57.089245 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.089164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk" (OuterVolumeSpecName: "kube-api-access-hz6pk") pod "59fd33d9-b39b-452c-ab22-dd881fc613cf" (UID: "59fd33d9-b39b-452c-ab22-dd881fc613cf"). InnerVolumeSpecName "kube-api-access-hz6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:00:57.188475 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.188432    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fd33d9-b39b-452c-ab22-dd881fc613cf-maas-api-tls\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:57.188475 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.188470    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fd33d9-b39b-452c-ab22-dd881fc613cf-openshift-service-ca\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:57.188475 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.188482    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz6pk\" (UniqueName: \"kubernetes.io/projected/59fd33d9-b39b-452c-ab22-dd881fc613cf-kube-api-access-hz6pk\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:00:57.309555 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.309512    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 16:00:57.315418 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:57.315391    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-f65ca6-685b7df7c7-52ch8"]
Jul 17 16:00:58.241315 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:00:58.241281    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59fd33d9-b39b-452c-ab22-dd881fc613cf" path="/var/lib/kubelet/pods/59fd33d9-b39b-452c-ab22-dd881fc613cf/volumes"
Jul 17 16:01:11.882905 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:11.882869    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 16:01:11.883412 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:11.883106    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" containerID="cri-o://6e865d012989d92f7b0f35ceece6a8b3fe570503dd7c0e017fe7e958b4c81844" gracePeriod=30
Jul 17 16:01:13.583430 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:13.583388    2566 patch_prober.go:28] interesting pod/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 16:01:13.583853 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:13.583481    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 16:01:17.086490 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.086457    2566 generic.go:356] "Generic (PLEG): container finished" podID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerID="6e865d012989d92f7b0f35ceece6a8b3fe570503dd7c0e017fe7e958b4c81844" exitCode=0
Jul 17 16:01:17.086979 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.086506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerDied","Data":"6e865d012989d92f7b0f35ceece6a8b3fe570503dd7c0e017fe7e958b4c81844"}
Jul 17 16:01:17.086979 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.086559    2566 scope.go:117] "RemoveContainer" containerID="9985fee20a8fa92c2a58bf946db3109e014d582215aae582cb21d78d0b9cdb50"
Jul 17 16:01:17.142504 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.142474    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 16:01:17.175429 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175429 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9p7k\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175665 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175983 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert\") pod \"f55d1709-e81a-4273-855b-cd685ca6dab8\" (UID: \"f55d1709-e81a-4273-855b-cd685ca6dab8\") "
Jul 17 16:01:17.175983 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.175892    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:01:17.176123 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.176074    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-certs\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.176178 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.176152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:01:17.176178 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.176159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:01:17.176462 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.176396    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 16:01:17.177127 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.177100    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:01:17.178113 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.178086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 16:01:17.178243 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.178131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 16:01:17.178432 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.178411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:01:17.178567 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.178546    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k" (OuterVolumeSpecName: "kube-api-access-m9p7k") pod "f55d1709-e81a-4273-855b-cd685ca6dab8" (UID: "f55d1709-e81a-4273-855b-cd685ca6dab8"). InnerVolumeSpecName "kube-api-access-m9p7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 16:01:17.277323 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277292    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-credential-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277323 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277320    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55d1709-e81a-4273-855b-cd685ca6dab8-istiod-ca-cert\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277323 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277331    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-data\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277571 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277338    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-token\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277571 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277346    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-workload-socket\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277571 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277354    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p7k\" (UniqueName: \"kubernetes.io/projected/f55d1709-e81a-4273-855b-cd685ca6dab8-kube-api-access-m9p7k\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277571 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277372    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-podinfo\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:17.277571 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:17.277380    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55d1709-e81a-4273-855b-cd685ca6dab8-istio-envoy\") on node \"ip-10-0-128-207.ec2.internal\" DevicePath \"\""
Jul 17 16:01:18.092214 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.092186    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"
Jul 17 16:01:18.092675 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.092181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf" event={"ID":"f55d1709-e81a-4273-855b-cd685ca6dab8","Type":"ContainerDied","Data":"96ab678cf91c520a8431413817b6903abd7bf51780fb1b7273e38b6f42282535"}
Jul 17 16:01:18.092675 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.092323    2566 scope.go:117] "RemoveContainer" containerID="6e865d012989d92f7b0f35ceece6a8b3fe570503dd7c0e017fe7e958b4c81844"
Jul 17 16:01:18.114868 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.114832    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 16:01:18.119553 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.119523    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f65ca6-openshift-default-6bf68d674c-nqlrf"]
Jul 17 16:01:18.241130 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.241090    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" path="/var/lib/kubelet/pods/f55d1709-e81a-4273-855b-cd685ca6dab8/volumes"
Jul 17 16:01:18.804247 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.804222    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb95c7566-zxvfw_734db4ee-d1c7-4e56-aa75-e16ee5168fa1/copy-manifests/0.log"
Jul 17 16:01:18.812621 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:18.812569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb95c7566-zxvfw_734db4ee-d1c7-4e56-aa75-e16ee5168fa1/manager/0.log"
Jul 17 16:01:20.520769 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:20.520728    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5hrr8_53953c56-7fd3-4dd9-9702-a2dac05597fc/postgres/0.log"
Jul 17 16:01:21.943467 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:21.943407    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sv9jd_ab3e808c-304a-4461-8a64-64323dc1042c/manager/0.log"
Jul 17 16:01:23.372313 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:23.372282    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c457bb55-wq4lj_0db9cbb1-715b-49a2-b994-76bd4b28959d/kube-auth-proxy/0.log"
Jul 17 16:01:24.324278 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:24.324242    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-prbb6_3b8adda1-42c8-4528-aa7b-f136715560e3/main/0.log"
Jul 17 16:01:24.778519 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:24.778494    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-49ctd_f11308d2-2c24-448d-9412-aaf3aedc2158/main/0.log"
Jul 17 16:01:30.085111 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085067    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvc44/must-gather-4gxhd"]
Jul 17 16:01:30.085640 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085618    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" containerName="maas-api"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085644    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" containerName="maas-api"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085656    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085672    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085695    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085703    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085713    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fd33d9-b39b-452c-ab22-dd881fc613cf" containerName="maas-api"
Jul 17 16:01:30.085731 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085720    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fd33d9-b39b-452c-ab22-dd881fc613cf" containerName="maas-api"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085740    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085766    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c6b7ff-c111-4990-b24b-24649479e17c" containerName="main"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085774    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c6b7ff-c111-4990-b24b-24649479e17c" containerName="main"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085870    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085881    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6ad586-d766-46fa-bde2-25e68c67f7e4" containerName="maas-api"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085889    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085897    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c6b7ff-c111-4990-b24b-24649479e17c" containerName="main"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085905    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55d1709-e81a-4273-855b-cd685ca6dab8" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.085916    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fd33d9-b39b-452c-ab22-dd881fc613cf" containerName="maas-api"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.086103    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.086119 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.086116    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.086714 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.086212    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c97511a-0849-4fa5-8b54-7904edfcb02e" containerName="istio-proxy"
Jul 17 16:01:30.089545 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.089520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.097157 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.097039    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvc44\"/\"openshift-service-ca.crt\""
Jul 17 16:01:30.097384 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.097366    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fvc44\"/\"default-dockercfg-txbgf\""
Jul 17 16:01:30.097516 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.097492    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvc44\"/\"kube-root-ca.crt\""
Jul 17 16:01:30.099959 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.099938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvc44/must-gather-4gxhd"]
Jul 17 16:01:30.198909 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.198821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-must-gather-output\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.198909 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.198900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4mz\" (UniqueName: \"kubernetes.io/projected/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-kube-api-access-tg4mz\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.299429 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.299389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4mz\" (UniqueName: \"kubernetes.io/projected/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-kube-api-access-tg4mz\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.299630 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.299478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-must-gather-output\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.299767 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.299752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-must-gather-output\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.308889 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.308858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4mz\" (UniqueName: \"kubernetes.io/projected/2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe-kube-api-access-tg4mz\") pod \"must-gather-4gxhd\" (UID: \"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe\") " pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.406620 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.406520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvc44/must-gather-4gxhd"
Jul 17 16:01:30.553990 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.553963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvc44/must-gather-4gxhd"]
Jul 17 16:01:30.556328 ip-10-0-128-207 kubenswrapper[2566]: W0717 16:01:30.556296    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d3cd9a0_d4b3_49ef_8b73_435fecc5a2fe.slice/crio-41ba8db45eea6725014681cac82aacd995dd613f449b3c153069407ac6b861bc WatchSource:0}: Error finding container 41ba8db45eea6725014681cac82aacd995dd613f449b3c153069407ac6b861bc: Status 404 returned error can't find the container with id 41ba8db45eea6725014681cac82aacd995dd613f449b3c153069407ac6b861bc
Jul 17 16:01:30.558583 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:30.558567    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 16:01:31.154132 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:31.154091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvc44/must-gather-4gxhd" event={"ID":"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe","Type":"ContainerStarted","Data":"41ba8db45eea6725014681cac82aacd995dd613f449b3c153069407ac6b861bc"}
Jul 17 16:01:32.165986 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:32.165124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvc44/must-gather-4gxhd" event={"ID":"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe","Type":"ContainerStarted","Data":"62197553e5e2551818de10dcd7c9a245912cd1ab72ee9b555632aec2ecc8655a"}
Jul 17 16:01:32.165986 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:32.165180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvc44/must-gather-4gxhd" event={"ID":"2d3cd9a0-d4b3-49ef-8b73-435fecc5a2fe","Type":"ContainerStarted","Data":"c32d630aab6b17b33aa587b9addcd34360312628d772039b2345197e9a5827ad"}
Jul 17 16:01:32.182654 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:32.182590    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvc44/must-gather-4gxhd" podStartSLOduration=1.343285595 podStartE2EDuration="2.182567933s" totalImagesPullingTime="839.282338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 16:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:01:32.182321995 +0000 UTC m=+3142.457470311" watchObservedRunningTime="2026-07-17 16:01:32.182567933 +0000 UTC m=+3142.457716255"
Jul 17 16:01:33.235545 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:33.235514    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-smxd9_da5436cb-fb18-4ba3-a0d2-f4db7dcc7f39/global-pull-secret-syncer/0.log"
Jul 17 16:01:33.331572 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:33.331540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f4pvj_a440e36a-8a41-4911-94e9-1c1ddb6e3eaa/konnectivity-agent/0.log"
Jul 17 16:01:33.392114 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:33.392031    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-207.ec2.internal_2da7476cfcbe41002bd2b6f5093090eb/haproxy/0.log"
Jul 17 16:01:38.492531 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:38.492492    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sv9jd_ab3e808c-304a-4461-8a64-64323dc1042c/manager/0.log"
Jul 17 16:01:40.837899 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:40.837757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75vzx_71506a20-0b90-4314-a750-f44ebb0a855b/kube-state-metrics/0.log"
Jul 17 16:01:40.866705 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:40.866667    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75vzx_71506a20-0b90-4314-a750-f44ebb0a855b/kube-rbac-proxy-main/0.log"
Jul 17 16:01:40.893649 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:40.893613    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75vzx_71506a20-0b90-4314-a750-f44ebb0a855b/kube-rbac-proxy-self/0.log"
Jul 17 16:01:40.941051 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:40.941019    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bff5fbbbc-lk6jj_8c64b493-6007-4d6f-8f38-a31c390af8a3/metrics-server/0.log"
Jul 17 16:01:41.021283 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.021253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dg4kt_09d5b15f-e7f4-490a-9713-452b2ddf077c/node-exporter/0.log"
Jul 17 16:01:41.053483 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.053459    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dg4kt_09d5b15f-e7f4-490a-9713-452b2ddf077c/kube-rbac-proxy/0.log"
Jul 17 16:01:41.089440 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.089293    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dg4kt_09d5b15f-e7f4-490a-9713-452b2ddf077c/init-textfile/0.log"
Jul 17 16:01:41.314268 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.314242    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-mgffm_2b11a47b-d852-4e1a-a881-7ea8050c6562/kube-rbac-proxy-main/0.log"
Jul 17 16:01:41.344094 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.343986    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-mgffm_2b11a47b-d852-4e1a-a881-7ea8050c6562/kube-rbac-proxy-self/0.log"
Jul 17 16:01:41.371716 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.371681    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-mgffm_2b11a47b-d852-4e1a-a881-7ea8050c6562/openshift-state-metrics/0.log"
Jul 17 16:01:41.425048 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.425016    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/prometheus/0.log"
Jul 17 16:01:41.452749 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.452722    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/config-reloader/0.log"
Jul 17 16:01:41.480958 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.480930    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/thanos-sidecar/0.log"
Jul 17 16:01:41.506356 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.506325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/kube-rbac-proxy-web/0.log"
Jul 17 16:01:41.535189 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.535161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/kube-rbac-proxy/0.log"
Jul 17 16:01:41.563381 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.563351    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/kube-rbac-proxy-thanos/0.log"
Jul 17 16:01:41.589799 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.589772    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_29653e64-77b4-43fe-94bc-2d6d951d13da/init-config-reloader/0.log"
Jul 17 16:01:41.651814 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.651736    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-jq9jm_33b6c420-8e69-4f03-87fd-7d6f37dfb25f/prometheus-operator/0.log"
Jul 17 16:01:41.677863 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.677794    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-jq9jm_33b6c420-8e69-4f03-87fd-7d6f37dfb25f/kube-rbac-proxy/0.log"
Jul 17 16:01:41.839074 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.839026    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/thanos-query/0.log"
Jul 17 16:01:41.864710 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.864686    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/kube-rbac-proxy-web/0.log"
Jul 17 16:01:41.890950 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.890802    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/kube-rbac-proxy/0.log"
Jul 17 16:01:41.924656 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.924569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/prom-label-proxy/0.log"
Jul 17 16:01:41.950587 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.950552    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/kube-rbac-proxy-rules/0.log"
Jul 17 16:01:41.983694 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:41.983661    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7778f65b86-55grl_446073c0-4064-4c2f-a1ad-bcc0c91aec4f/kube-rbac-proxy-metrics/0.log"
Jul 17 16:01:43.878248 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:43.878213    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"]
Jul 17 16:01:43.884333 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:43.884304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:43.891974 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:43.891928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"]
Jul 17 16:01:44.053265 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.053228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-podres\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.053535 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.053518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-lib-modules\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.053723 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.053704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-proc\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.053857 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.053843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2bp\" (UniqueName: \"kubernetes.io/projected/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-kube-api-access-gx2bp\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.054010 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.053997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-sys\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155277 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-sys\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155277 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-podres\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-lib-modules\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-sys\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-proc\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2bp\" (UniqueName: \"kubernetes.io/projected/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-kube-api-access-gx2bp\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-proc\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-podres\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.155527 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.155470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-lib-modules\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.164104 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.164075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2bp\" (UniqueName: \"kubernetes.io/projected/92d8a3e5-d017-41bd-b3e8-489db7d4bdf2-kube-api-access-gx2bp\") pod \"perf-node-gather-daemonset-s6ppc\" (UID: \"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2\") " pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.198324 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.197985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:44.390874 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.388475    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"]
Jul 17 16:01:44.561461 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.561432    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cd6b847d-mst57_c1aa13de-0ddb-4d14-af86-cceabc6fd8a3/console/0.log"
Jul 17 16:01:44.643163 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:44.643137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-rnd2q_b3e09dc3-9031-43cb-bd47-ca5066bc0579/download-server/0.log"
Jul 17 16:01:45.264717 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:45.264675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc" event={"ID":"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2","Type":"ContainerStarted","Data":"d635037d570d9ea03961c0b9083f6951ad5eed888fab0737585bcc1ae2ad7a98"}
Jul 17 16:01:45.265243 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:45.264787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:45.265243 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:45.264807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc" event={"ID":"92d8a3e5-d017-41bd-b3e8-489db7d4bdf2","Type":"ContainerStarted","Data":"89381b3f933fd0a277ea023d2d56f8b744c1392e74b92a216f5522f3e10e259c"}
Jul 17 16:01:45.283661 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:45.283549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc" podStartSLOduration=2.283526071 podStartE2EDuration="2.283526071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 16:01:45.282378911 +0000 UTC m=+3155.557527238" watchObservedRunningTime="2026-07-17 16:01:45.283526071 +0000 UTC m=+3155.558674393"
Jul 17 16:01:46.456121 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:46.456091    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8892_03b96b1e-5919-432e-a238-47678e66b248/dns/0.log"
Jul 17 16:01:46.483405 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:46.483378    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r8892_03b96b1e-5919-432e-a238-47678e66b248/kube-rbac-proxy/0.log"
Jul 17 16:01:46.651626 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:46.651596    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hlbt_02bbe719-e77d-43ec-b9e4-9737732089aa/dns-node-resolver/0.log"
Jul 17 16:01:47.382785 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:47.382752    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54d59b885f-j4dc5_f24e56c6-84c5-475d-acff-75f6040ca00b/registry/0.log"
Jul 17 16:01:47.446941 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:47.446907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7tx5z_e9e5939e-d8b3-4860-857d-e7d6b6726d31/node-ca/0.log"
Jul 17 16:01:48.738533 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:48.738502    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c457bb55-wq4lj_0db9cbb1-715b-49a2-b994-76bd4b28959d/kube-auth-proxy/0.log"
Jul 17 16:01:49.645539 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:49.645495    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-44t9v_6f2ea865-60f9-49d9-91ac-4acd4b2c685b/serve-healthcheck-canary/0.log"
Jul 17 16:01:50.438884 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:50.438855    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwst8_6eea7d47-c619-40e3-be9f-553b8df97479/kube-rbac-proxy/0.log"
Jul 17 16:01:50.465076 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:50.465038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwst8_6eea7d47-c619-40e3-be9f-553b8df97479/exporter/0.log"
Jul 17 16:01:50.500724 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:50.500686    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zwst8_6eea7d47-c619-40e3-be9f-553b8df97479/extractor/0.log"
Jul 17 16:01:51.287658 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:51.287624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fvc44/perf-node-gather-daemonset-s6ppc"
Jul 17 16:01:53.571195 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:53.571122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb95c7566-zxvfw_734db4ee-d1c7-4e56-aa75-e16ee5168fa1/manager/0.log"
Jul 17 16:01:53.597395 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:53.597369    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb95c7566-zxvfw_734db4ee-d1c7-4e56-aa75-e16ee5168fa1/copy-manifests/0.log"
Jul 17 16:01:54.663642 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:54.663605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c457bb55-wq4lj_0db9cbb1-715b-49a2-b994-76bd4b28959d/kube-auth-proxy/0.log"
Jul 17 16:01:55.012822 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:55.012794    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76c68485bc-sbqdb_ec5e6ff9-705f-421c-b191-eae8ece4826c/manager/0.log"
Jul 17 16:01:56.263170 ip-10-0-128-207 kubenswrapper[2566]: I0717 16:01:56.263140    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c457bb55-wq4lj_0db9cbb1-715b-49a2-b994-76bd4b28959d/kube-auth-proxy/0.log"