Jul 27 20:15:38.227113 ip-10-0-143-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 20:15:38.227124 ip-10-0-143-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 20:15:38.227134 ip-10-0-143-74 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 20:15:38.227424 ip-10-0-143-74 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 20:15:48.351937 ip-10-0-143-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 20:15:48.351952 ip-10-0-143-74 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 533e82557a444e8a9984c5026cab4583 --
Jul 27 20:17:47.028140 ip-10-0-143-74 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:17:47.451295 ip-10-0-143-74 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:17:47.451295 ip-10-0-143-74 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:17:47.451295 ip-10-0-143-74 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:17:47.451295 ip-10-0-143-74 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:17:47.451295 ip-10-0-143-74 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:17:47.453369 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.453252    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:17:47.456934 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456916    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.456934 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456932    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456936    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456939    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456942    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456947    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456951    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456953    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456956    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456959    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456961    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456964    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456966    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456969    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456972    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456974    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456976    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456984    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456986    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456989    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.457001 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456991    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456994    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456996    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.456999    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457001    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457005    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457009    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457011    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457015    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457018    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457020    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457023    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457025    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457028    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457030    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457032    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457035    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457037    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457040    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457043    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.457451 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457046    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457048    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457051    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457053    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457055    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457058    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457060    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457062    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457065    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457067    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457069    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457071    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457074    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457076    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457079    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457082    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457085    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457087    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457089    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.457911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457092    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457094    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457097    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457099    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457102    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457104    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457106    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457109    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457111    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457113    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457116    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457118    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457121    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457123    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457126    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457129    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457133    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457136    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457138    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.458336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457141    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457143    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457146    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457148    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457151    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457153    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457155    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457157    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457160    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457163    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457166    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457168    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457170    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457173    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457175    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457178    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457180    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457182    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457185    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457187    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457189    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.458759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457895    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457902    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457905    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457907    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457911    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457916    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457918    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457922    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457924    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457927    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457930    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457932    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457935    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457937    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457940    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457942    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457945    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457947    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457950    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457953    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.459234 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457955    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457958    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457961    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457963    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457965    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457968    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457970    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457972    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457975    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457977    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457979    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457982    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457984    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457986    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457989    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457991    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457993    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457995    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.457998    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458001    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458003    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.459689 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458006    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458009    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458011    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458014    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458017    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458019    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458021    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458024    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458027    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458029    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458032    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458034    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458037    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458039    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458052    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458056    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458059    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458061    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458064    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458066    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.460167 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458069    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458071    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458074    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458076    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458078    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458080    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458083    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458085    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458087    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458090    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458094    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458097    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458100    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458103    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458105    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458108    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458111    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458114    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458116    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.460632 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458118    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458122    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458125    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458128    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458131    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458134    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458136    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458139    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458142    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458144    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458147    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458149    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458152    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458154    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458156    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458158    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458161    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458163    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458165    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.461085 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458538    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458543    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458546    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458548    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458551    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458554    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458557    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458559    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458562    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458564    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458567    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458569    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458571    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458573    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458576    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458578    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458581    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458583    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458586    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458588    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.461514 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458591    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458594    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458596    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458600    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458603    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458606    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458608    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458611    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458613    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458615    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458618    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458620    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458623    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458625    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458627    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458629    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458632    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458634    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458637    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.462003 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458641    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458644    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458647    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458650    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458652    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458654    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458656    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458659    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458661    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458664    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458666    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458668    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458671    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458673    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458675    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458678    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458680    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458682    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458684    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458687    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.462530 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458689    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458692    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458694    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458697    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458699    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458701    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458703    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458705    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458708    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458710    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458712    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458715    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458721    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458723    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458732    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458734    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458736    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458739    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458741    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458743    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.463009 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458746    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458749    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458752    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458755    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458757    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458760    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458764    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458768    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458771    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458773    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458776    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458778    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458781    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458784    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458786    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458789    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458791    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458793    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458796    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.458798    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.463488 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458896    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458917    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458927    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458932    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458938    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458941    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458945    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458949    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458952    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458955    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458958    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458961    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458964    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458967    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458970    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458973    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458976    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458978    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458982    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458984    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458987    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458991    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458994    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.458997    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:17:47.463971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459000    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459003    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459007    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459009    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459012    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459015    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459018    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459020    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459024    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459027    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459030    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459032    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459036    2560 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459040    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459045    2560 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459047    2560 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459050    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459053    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459056    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459059    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459062    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459065    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459068    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459071    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459074    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:17:47.464498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459076    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459079    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459082    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459085    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459088    2560 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459091    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459094    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459097    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459100    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459103    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459105    2560 flags.go:64] FLAG: --help="false"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459108    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459111    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459115    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459118    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459121    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459124    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459127    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459130    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459133    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459136    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459139    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459142    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459145    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:17:47.465110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459148    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459150    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459153    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459156    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459159    2560 flags.go:64] FLAG: --lock-file=""
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459162    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459164    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459167    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459172    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459175    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459177    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459180    2560 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459183    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459186    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459188    2560 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459205    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459212    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459215    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459218    2560 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459221    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459224    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459227    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459229    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459232    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459235    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:17:47.465683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459238    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459245    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459249    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459252    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459255    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459259    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459265    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459267    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459270    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459273    2560 flags.go:64] FLAG: --port="10250"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459276    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459279    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04810afa82f146e9f"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459282    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459285    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459288    2560 flags.go:64] FLAG: --register-node="true"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459290    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459293    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459296    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459299    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459301    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459304    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459307    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459310    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459312    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459315    2560 flags.go:64] FLAG: --runonce="false"
Jul 27 20:17:47.466275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459317    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459320    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459323    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459325    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459328    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459331    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459333    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459336    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459338    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459342    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459345    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459347    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459350    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459354    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459356    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459361    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459364    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459367    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459371    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459375    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459377    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459380    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459383    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459385    2560 flags.go:64] FLAG: --v="2"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459389    2560 flags.go:64] FLAG: --version="false"
Jul 27 20:17:47.466895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459393    2560 flags.go:64] FLAG: --vmodule=""
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459397    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.459400    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459517    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459520    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459523    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459526    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459529    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459532    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459534    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459537    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459539    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459542    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459545    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459548    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459551    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459555    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459558    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459561    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459564    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.467448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459567    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459570    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459573    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459575    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459578    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459581    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459585    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459588    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459591    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459593    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459596    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459598    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459601    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459603    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459606    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459608    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459610    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459613    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459615    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.467911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459618    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459620    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459622    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459625    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459627    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459630    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459632    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459634    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459637    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459640    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459643    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459645    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459647    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459650    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459652    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459655    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459657    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459659    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459662    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459665    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.468350 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459668    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459670    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459672    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459675    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459677    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459680    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459682    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459685    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459687    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459689    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459692    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459694    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459698    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459701    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459704    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459706    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459709    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459712    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459714    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.468911 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459717    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459719    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459732    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459735    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459738    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459740    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459743    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459745    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459747    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459750    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459753    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459755    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459759    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459761    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459764    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459766    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459768    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459771    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459773    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459775    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.469336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459778    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459780    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459783    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.459785    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.460303    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.466944    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.466960    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.468056    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.468063    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468118    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468124    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468127    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468130    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468133    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468136    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.469798 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468139    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468141    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468143    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468146    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468148    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468150    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468153    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468156    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468158    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468161    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468163    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468166    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468170    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468173    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468176    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468178    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468181    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468183    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468185    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.470157 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468188    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468190    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468193    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468195    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468199    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468202    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468204    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468207    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468209    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468212    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468214    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468217    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468219    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468222    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468224    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468226    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468228    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468231    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468233    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.470580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468236    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468238    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468240    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468243    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468245    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468247    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468250    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468253    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468255    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468257    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468260    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468262    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468264    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468267    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468271    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468274    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468277    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468281    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468284    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.471046 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468287    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468290    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468293    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468296    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468298    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468301    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468303    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468305    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468308    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468310    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468313    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468315    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468317    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468319    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468321    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468324    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468326    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468329    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468331    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468333    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.471501 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468336    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468338    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468341    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468343    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468346    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468348    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468351    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468354    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468357    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468359    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468362    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468364    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468368    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468371    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468373    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.471999 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468375    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.468380    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468490    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468494    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468497    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468499    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468501    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468504    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468506    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468509    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468511    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468513    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468516    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468518    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468521    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:17:47.472378 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468523    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468526    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468528    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468531    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468533    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468536    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468538    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468540    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468543    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468546    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468549    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468551    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468553    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468555    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468558    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468561    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468564    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468568    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468570    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468573    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:17:47.472875 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468575    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468578    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468580    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468582    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468585    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468587    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468589    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468591    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468593    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468596    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468598    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468600    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468602    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468605    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468608    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468610    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468612    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468615    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468617    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468620    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:17:47.473336 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468622    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468624    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468627    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468629    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468631    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468634    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468636    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468638    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468641    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468643    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468645    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468648    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468650    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468652    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468654    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468657    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468660    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468663    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468666    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:17:47.473994 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468669    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468671    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468673    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468676    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468678    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468680    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468683    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468686    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468689    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468692    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468694    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468697    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468700    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468702    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468705    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468708    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468710    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468713    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468715    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468717    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:17:47.474765 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468720    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468722    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468726    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468730    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468733    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468736    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:47.468738    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.468748    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.468898    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.470884    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.472289    2560 server.go:1019] "Starting client certificate rotation"
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.472393    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.472442    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:17:47.475380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.474955    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:17:47.479077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.479054    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:17:47.487456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.487438    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:17:47.491675 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.491656    2560 log.go:25] "Validated CRI v1 image API"
Jul 27 20:17:47.492480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.492456    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:17:47.496138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.496107    2560 fs.go:135] Filesystem UUIDs: map[741207cb-9e6b-498a-8aac-0cdc59d99a68:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ea45cc2f-4a24-4024-811e-4c3a53b72e96:/dev/nvme0n1p3]
Jul 27 20:17:47.496244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.496136    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:17:47.500966 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.500847    2560 manager.go:216] Machine: {Timestamp:2026-07-27 20:17:47.499676399 +0000 UTC m=+0.349443217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104766 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23f3daff3cd11f1bf306627fd01170 SystemUUID:ec23f3da-ff3c-d11f-1bf3-06627fd01170 BootID:533e8255-7a44-4e8a-9984-c5026cab4583 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:be:05:e8:af:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:be:05:e8:af:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:d2:8d:b6:ce:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:17:47.500966 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.500954    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:17:47.501105 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.501036    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:17:47.501879 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.501862    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:17:47.503955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.503924    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:17:47.504095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.503958    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:17:47.504862 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.504853    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:17:47.504893 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.504865    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:17:47.504893 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.504872    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:17:47.505629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.505620    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:17:47.506372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.506363    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:17:47.508173 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508162    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:17:47.508291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508282    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:17:47.508333 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508326    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:17:47.508362 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508342    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:17:47.508362 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508356    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:17:47.508429 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508365    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:17:47.508429 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508374    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:17:47.508838 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508824    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:17:47.508887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.508846    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:17:47.511489 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.511473    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:17:47.513408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.513395    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:17:47.513456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.513424    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:17:47.514985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.514971    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.514991    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.514997    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515002    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515010    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515016    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:17:47.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515023    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:17:47.515166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515028    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:17:47.515166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515035    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:17:47.515166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515040    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:17:47.515166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515053    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:17:47.515166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515058    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:17:47.515997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515989    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:17:47.515997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.515997    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:17:47.518974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.518954    2560 server.go:1268] "Started kubelet"
Jul 27 20:17:47.518974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.518969    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:17:47.519116 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.518983    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:17:47.519116 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.519030    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:17:47.519116 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.519093    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:17:47.519240 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.519147    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:17:47.519483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.519287    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:17:47.519841 ip-10-0-143-74 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:17:47.523782 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.523718    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:17:47.524534 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.524510    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrbjn"
Jul 27 20:17:47.525473 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.525434    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:17:47.525654 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.525640    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:17:47.529529 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.528563    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-74.ec2.internal.18c63ee7c85daa9e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-74.ec2.internal,UID:ip-10-0-143-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-74.ec2.internal,},FirstTimestamp:2026-07-27 20:17:47.518929566 +0000 UTC m=+0.368696384,LastTimestamp:2026-07-27 20:17:47.518929566 +0000 UTC m=+0.368696384,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-74.ec2.internal,}"
Jul 27 20:17:47.530097 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530082    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:17:47.530097 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530090    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:17:47.530203 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.530170    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.530780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530760    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vrbjn"
Jul 27 20:17:47.530918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530791    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:17:47.530918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530794    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:17:47.530918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530919    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:17:47.531086 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530929    2560 factory.go:55] Registering systemd factory
Jul 27 20:17:47.531086 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.530943    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:17:47.531178 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531156    2560 factory.go:153] Registering CRI-O factory
Jul 27 20:17:47.531178 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531169    2560 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:17:47.531270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531219    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:17:47.531270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531240    2560 factory.go:103] Registering Raw factory
Jul 27 20:17:47.531270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531254    2560 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:17:47.531724 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531712    2560 manager.go:349] Starting recovery of all containers
Jul 27 20:17:47.531798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531786    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:17:47.531798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.531799    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:17:47.539967 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.539938    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:47.542353 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.542315    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-74.ec2.internal\" not found" node="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.542576 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.542542    2560 manager.go:354] Recovery completed
Jul 27 20:17:47.547168 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.547152    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.549883 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.549862    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.549955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.549895    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.549955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.549909    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.550437 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.550423    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:17:47.550493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.550435    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:17:47.550493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.550445    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.550493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.550455    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:17:47.551276 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.551258    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.551343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.551283    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.551343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.551293    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.552131 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.552119    2560 policy_none.go:49] "None policy: Start"
Jul 27 20:17:47.552174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.552134    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:17:47.552174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.552145    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:17:47.553059 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.553050    2560 policy_none.go:47] "Start"
Jul 27 20:17:47.588678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.588405    2560 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:17:47.588678 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.588642    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:17:47.588678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.588656    2560 server.go:87] "Starting device plugin registration server"
Jul 27 20:17:47.588929 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.588918    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:17:47.588980 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.588929    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:17:47.589047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.589026    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:17:47.589150 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.589109    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:17:47.589150 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.589118    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:17:47.589685 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.589666    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:17:47.589751 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.589716    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.690033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.690004    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.690996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.690982    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.691053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.691010    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.691053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.691023    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.691053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.691049    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.698258 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.698241    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.698335 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.698264    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-74.ec2.internal\": node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.707904 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.707879    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.718762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.718730    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:17:47.719915 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.719893    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:17:47.720022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.719925    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:17:47.720022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.719945    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:17:47.720022 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.719987    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:17:47.722697 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.722672    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:47.808558 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.808523    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.820675 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.820646    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"]
Jul 27 20:17:47.820750 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.820728    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.822250 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.822235    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.822340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.822261    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.822340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.822270    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.823316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823305    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.823436 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.823436 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823432    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.823989 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823973    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.824069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823973    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.824069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.823999    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.824069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.824009    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.824069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.824014    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.824069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.824022    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.825118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.825106    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.825162 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.825127    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:17:47.825755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.825740    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:17:47.825822 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.825778    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:17:47.825822 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.825791    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:17:47.834155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.834135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.834241 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.834160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.834241 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.834177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84ce0d9219c2a58af44c132cfd94a88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-74.ec2.internal\" (UID: \"f84ce0d9219c2a58af44c132cfd94a88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.857667 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.857600    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-74.ec2.internal\" not found" node="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.861865 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.861851    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-74.ec2.internal\" not found" node="ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.908704 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:47.908680    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:47.935052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.935125 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.935125 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84ce0d9219c2a58af44c132cfd94a88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-74.ec2.internal\" (UID: \"f84ce0d9219c2a58af44c132cfd94a88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.935125 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.935224 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/198dc54988f9644415b48f7e98d8dd24-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal\" (UID: \"198dc54988f9644415b48f7e98d8dd24\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:47.935224 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:47.935107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f84ce0d9219c2a58af44c132cfd94a88-config\") pod \"kube-apiserver-proxy-ip-10-0-143-74.ec2.internal\" (UID: \"f84ce0d9219c2a58af44c132cfd94a88\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:48.009280 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.009242    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:48.109761 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.109693    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:48.160204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.160181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:48.164887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.164863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:48.210761 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.210728    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:48.311187 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.311155    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-74.ec2.internal\" not found"
Jul 27 20:17:48.402509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.402428    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:48.431065 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.431028    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:48.441577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.441556    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:17:48.443609 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.443591    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal"
Jul 27 20:17:48.452135 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.452112    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:17:48.472324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.472296    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:17:48.472476 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.472452    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:17:48.472523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.472451    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:17:48.472523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.472457    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:17:48.488041 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.488015    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:48.509159 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.509131    2560 apiserver.go:52] "Watching apiserver"
Jul 27 20:17:48.516854 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.516823    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:17:48.517923 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.517903    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-8gdvn","openshift-multus/multus-additional-cni-plugins-dq6h9","openshift-multus/multus-j5qpn","openshift-multus/network-metrics-daemon-gvsbh","openshift-network-diagnostics/network-check-target-t7llv","openshift-ovn-kubernetes/ovnkube-node-nk775","kube-system/global-pull-secret-syncer-npmtt","kube-system/konnectivity-agent-z5hqg","openshift-cluster-node-tuning-operator/tuned-p9v6l","openshift-dns/node-resolver-crkxs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal","openshift-network-operator/iptables-alerter-9486k","kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"]
Jul 27 20:17:48.519667 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.519649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.519777 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.519723    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:48.520845 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.520829    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.522997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.522978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.523868 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.523845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:17:48.523975 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.523912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:17:48.524261 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.524241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4qrw9\""
Jul 27 20:17:48.524349 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.524342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.524706 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.524682    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:48.525155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.525128    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r7cj7\""
Jul 27 20:17:48.525220 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.525164    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:17:48.525220 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.525132    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.525484 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.525471    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:17:48.525857 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.525843    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.526316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.526304    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:48.526394 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.526362    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:48.527756 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.527740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.529062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.529043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.529714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.529697    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:17:48.529714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.529704    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:17:48.529916 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.529749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-462gb\""
Jul 27 20:17:48.529916 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.529870    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:17:48.530180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.530167    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:17:48.530266 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.530253    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.530401 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.530384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.530964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.530946    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.530964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.530960    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:17:48.531103 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.531045    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.531253 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.531239    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mhpbk\""
Jul 27 20:17:48.531318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.531306    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.532299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.532283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.532485 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.532469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:17:48.532733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.532712    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cvq2r\""
Jul 27 20:17:48.532860 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.532756    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.532970 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.532957    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.533671 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.533658    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.534350 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.534332    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:17:48.534450 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.534336    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xfn5\""
Jul 27 20:17:48.534801 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.534789    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:17:48.535166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.535152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.535921 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.535641    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:12:47 +0000 UTC" deadline="2028-03-08 20:00:21.035823857 +0000 UTC"
Jul 27 20:17:48.535921 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.535679    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14159h42m32.500148674s"
Jul 27 20:17:48.536076 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.536063    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pq5r5\""
Jul 27 20:17:48.536304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.536288    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.536304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.536299    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.536549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.536534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.537566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.537550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:17:48.537999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.537981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.538119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.538201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-node-log\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovn-node-metrics-cert\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-tmp\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-dbus\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-multus\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69q92\" (UniqueName: \"kubernetes.io/projected/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-kube-api-access-69q92\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-systemd-units\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-script-lib\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-sys\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-kubelet-config\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cnibin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-netd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-modprobe-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlpg\" (UniqueName: \"kubernetes.io/projected/7d905ce2-dab1-4209-afa0-8107f39f9124-kube-api-access-mtlpg\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-conf-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-os-release\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-system-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.538631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cni-binary-copy\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-k8s-cni-cncf-io\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538714    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-kubelet\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-etc-kubernetes\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-run\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mg5b\" (UniqueName: \"kubernetes.io/projected/1a0b6962-d577-4475-b9c6-c098e90435b7-kube-api-access-5mg5b\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-var-lib-kubelet\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538887    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d905ce2-dab1-4209-afa0-8107f39f9124-serviceca\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsmrb\""
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysconfig\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d905ce2-dab1-4209-afa0-8107f39f9124-host\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538971    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.538981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61fb11b7-fbab-4120-87a5-e4dce9efc13c-agent-certs\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-os-release\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539034    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-socket-dir-parent\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-systemd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539194 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-log-socket\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-binary-copy\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-daemon-config\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-slash\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-netns\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l8fvh\""
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprp8\" (UniqueName: \"kubernetes.io/projected/4e16b3df-cbd5-4241-99b8-e199148b68e0-kube-api-access-rprp8\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-lib-modules\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-system-cni-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61fb11b7-fbab-4120-87a5-e4dce9efc13c-konnectivity-ca\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-host\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-bin\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-env-overrides\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-kubernetes\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-tuned\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-multus-certs\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.539985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-kubelet\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-etc-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-conf\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-netns\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-ovn\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8vb\" (UniqueName: \"kubernetes.io/projected/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-kube-api-access-nv8vb\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-hostroot\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-var-lib-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqf5p\" (UniqueName: \"kubernetes.io/projected/1da0167a-bfd5-4791-a03c-52fc3795d410-kube-api-access-dqf5p\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cnibin\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-bin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-config\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.540471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-systemd\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.541031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.539980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.541184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.541040    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:17:48.560766 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.560744    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ksf44"
Jul 27 20:17:48.569920 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.569896    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ksf44"
Jul 27 20:17:48.631654 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.631630    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:17:48.639168 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.639125    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84ce0d9219c2a58af44c132cfd94a88.slice/crio-f4faf709f71049fb3ecff3e1b73cd4e2be152aceef4dc9392501936c4e40a536 WatchSource:0}: Error finding container f4faf709f71049fb3ecff3e1b73cd4e2be152aceef4dc9392501936c4e40a536: Status 404 returned error can't find the container with id f4faf709f71049fb3ecff3e1b73cd4e2be152aceef4dc9392501936c4e40a536
Jul 27 20:17:48.640186 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-bin\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.640304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-env-overrides\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.640304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-bin\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.640411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-kubernetes\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.640411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/763f05fe-0d04-49e8-896c-c5b095c221e5-host-slash\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.640411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.640411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-kubernetes\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.640411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-tuned\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/763f05fe-0d04-49e8-896c-c5b095c221e5-iptables-alerter-script\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdlg\" (UniqueName: \"kubernetes.io/projected/763f05fe-0d04-49e8-896c-c5b095c221e5-kube-api-access-ksdlg\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-multus-certs\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-kubelet\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-multus-certs\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.640631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-etc-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-kubelet\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-etc-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-conf\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dba724cb-8bb8-4f73-be21-ab451f82a6ff-hosts-file\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dba724cb-8bb8-4f73-be21-ab451f82a6ff-tmp-dir\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640819    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-netns\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-ovn\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8vb\" (UniqueName: \"kubernetes.io/projected/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-kube-api-access-nv8vb\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-hostroot\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-conf\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-var-lib-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqf5p\" (UniqueName: \"kubernetes.io/projected/1da0167a-bfd5-4791-a03c-52fc3795d410-kube-api-access-dqf5p\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.641095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.640985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cnibin\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-socket-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-bin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-config\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-systemd\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cnibin\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-env-overrides\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641371    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641457    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-systemd\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.641798 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.641763    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-var-lib-openvswitch\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.641859    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:49.141827808 +0000 UTC m=+1.991594636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-hostroot\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-netns\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.641938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-ovn\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-bin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-config\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-node-log\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovn-node-metrics-cert\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-node-log\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-tmp\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-dbus\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-multus\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysctl-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.642448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69q92\" (UniqueName: \"kubernetes.io/projected/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-kube-api-access-69q92\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-systemd-units\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-script-lib\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-sys\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-kubelet-config\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54b78\" (UniqueName: \"kubernetes.io/projected/dba724cb-8bb8-4f73-be21-ab451f82a6ff-kube-api-access-54b78\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cnibin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-netd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-modprobe-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlpg\" (UniqueName: \"kubernetes.io/projected/7d905ce2-dab1-4209-afa0-8107f39f9124-kube-api-access-mtlpg\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-cni-multus\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-conf-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cnibin\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-os-release\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-systemd-units\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-system-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cni-binary-copy\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-system-cni-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-k8s-cni-cncf-io\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-kubelet\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-etc-kubernetes\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-run\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mg5b\" (UniqueName: \"kubernetes.io/projected/1a0b6962-d577-4475-b9c6-c098e90435b7-kube-api-access-5mg5b\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-device-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-var-lib-kubelet\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d905ce2-dab1-4209-afa0-8107f39f9124-serviceca\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysconfig\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d905ce2-dab1-4209-afa0-8107f39f9124-host\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-cni-netd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.642993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-registration-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61fb11b7-fbab-4120-87a5-e4dce9efc13c-agent-certs\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-modprobe-d\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.643888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-os-release\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.642522    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-socket-dir-parent\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.643093    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:49.143078043 +0000 UTC m=+1.992844863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-systemd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-dbus\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-socket-dir-parent\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c95815a-2734-4e41-82bf-bd4d0f5883f2-kubelet-config\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643391    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-run-systemd\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovnkube-script-lib\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-var-lib-kubelet\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d905ce2-dab1-4209-afa0-8107f39f9124-host\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-run-k8s-cni-cncf-io\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-os-release\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-sys\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-host-var-lib-kubelet\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.644634 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-run\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-sysconfig\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d905ce2-dab1-4209-afa0-8107f39f9124-serviceca\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-log-socket\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-conf-dir\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-os-release\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-log-socket\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-etc-kubernetes\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-etc-tuned\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.643667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-cni-binary-copy\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-binary-copy\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-etc-selinux\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-daemon-config\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-slash\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-netns\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprp8\" (UniqueName: \"kubernetes.io/projected/4e16b3df-cbd5-4241-99b8-e199148b68e0-kube-api-access-rprp8\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e16b3df-cbd5-4241-99b8-e199148b68e0-ovn-node-metrics-cert\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-lib-modules\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.645347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-system-cni-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-run-netns\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61fb11b7-fbab-4120-87a5-e4dce9efc13c-konnectivity-ca\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-host\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-sys-fs\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6sx\" (UniqueName: \"kubernetes.io/projected/65425295-4d07-4177-a298-158626a08624-kube-api-access-7c6sx\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-lib-modules\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a0b6962-d577-4475-b9c6-c098e90435b7-tmp\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-system-cni-dir\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-cni-binary-copy\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a0b6962-d577-4475-b9c6-c098e90435b7-host\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e16b3df-cbd5-4241-99b8-e199148b68e0-host-slash\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.644914    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-multus-daemon-config\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.645017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61fb11b7-fbab-4120-87a5-e4dce9efc13c-konnectivity-ca\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.646189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.645607    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:17:48.646864 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.646016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61fb11b7-fbab-4120-87a5-e4dce9efc13c-agent-certs\") pod \"konnectivity-agent-z5hqg\" (UID: \"61fb11b7-fbab-4120-87a5-e4dce9efc13c\") " pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.652890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.652793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8vb\" (UniqueName: \"kubernetes.io/projected/3f2a7e81-83ab-41cc-b66d-c1ae1592ad14-kube-api-access-nv8vb\") pod \"multus-additional-cni-plugins-dq6h9\" (UID: \"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14\") " pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.653787 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.653578    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:17:48.653787 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.653604    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:17:48.653787 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.653617    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:48.653787 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:48.653682    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:49.153664777 +0000 UTC m=+2.003431584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:48.655765 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.655743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mg5b\" (UniqueName: \"kubernetes.io/projected/1a0b6962-d577-4475-b9c6-c098e90435b7-kube-api-access-5mg5b\") pod \"tuned-p9v6l\" (UID: \"1a0b6962-d577-4475-b9c6-c098e90435b7\") " pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.655985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.655968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlpg\" (UniqueName: \"kubernetes.io/projected/7d905ce2-dab1-4209-afa0-8107f39f9124-kube-api-access-mtlpg\") pod \"node-ca-8gdvn\" (UID: \"7d905ce2-dab1-4209-afa0-8107f39f9124\") " pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.656680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.656658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprp8\" (UniqueName: \"kubernetes.io/projected/4e16b3df-cbd5-4241-99b8-e199148b68e0-kube-api-access-rprp8\") pod \"ovnkube-node-nk775\" (UID: \"4e16b3df-cbd5-4241-99b8-e199148b68e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.656680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.656672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqf5p\" (UniqueName: \"kubernetes.io/projected/1da0167a-bfd5-4791-a03c-52fc3795d410-kube-api-access-dqf5p\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:48.656779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.656758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69q92\" (UniqueName: \"kubernetes.io/projected/5e7f7396-c4d5-4a5d-86c9-7a2e013fb365-kube-api-access-69q92\") pod \"multus-j5qpn\" (UID: \"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365\") " pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.723755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.723704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal" event={"ID":"f84ce0d9219c2a58af44c132cfd94a88","Type":"ContainerStarted","Data":"f4faf709f71049fb3ecff3e1b73cd4e2be152aceef4dc9392501936c4e40a536"}
Jul 27 20:17:48.733360 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.733335    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198dc54988f9644415b48f7e98d8dd24.slice/crio-c714df7c03e330ed61257045ee75883802299f07d38ed38389f95ba0a1b4ebbc WatchSource:0}: Error finding container c714df7c03e330ed61257045ee75883802299f07d38ed38389f95ba0a1b4ebbc: Status 404 returned error can't find the container with id c714df7c03e330ed61257045ee75883802299f07d38ed38389f95ba0a1b4ebbc
Jul 27 20:17:48.745179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745294 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54b78\" (UniqueName: \"kubernetes.io/projected/dba724cb-8bb8-4f73-be21-ab451f82a6ff-kube-api-access-54b78\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.745294 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-device-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745294 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-registration-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745294 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-etc-selinux\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-registration-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-sys-fs\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-device-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-sys-fs\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6sx\" (UniqueName: \"kubernetes.io/projected/65425295-4d07-4177-a298-158626a08624-kube-api-access-7c6sx\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/763f05fe-0d04-49e8-896c-c5b095c221e5-host-slash\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-etc-selinux\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/763f05fe-0d04-49e8-896c-c5b095c221e5-iptables-alerter-script\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/763f05fe-0d04-49e8-896c-c5b095c221e5-host-slash\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745465    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdlg\" (UniqueName: \"kubernetes.io/projected/763f05fe-0d04-49e8-896c-c5b095c221e5-kube-api-access-ksdlg\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dba724cb-8bb8-4f73-be21-ab451f82a6ff-hosts-file\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.745508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dba724cb-8bb8-4f73-be21-ab451f82a6ff-tmp-dir\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.745958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-socket-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dba724cb-8bb8-4f73-be21-ab451f82a6ff-hosts-file\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.745958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65425295-4d07-4177-a298-158626a08624-socket-dir\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.745958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dba724cb-8bb8-4f73-be21-ab451f82a6ff-tmp-dir\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.745958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.745930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/763f05fe-0d04-49e8-896c-c5b095c221e5-iptables-alerter-script\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.754404 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.754379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54b78\" (UniqueName: \"kubernetes.io/projected/dba724cb-8bb8-4f73-be21-ab451f82a6ff-kube-api-access-54b78\") pod \"node-resolver-crkxs\" (UID: \"dba724cb-8bb8-4f73-be21-ab451f82a6ff\") " pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.754855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.754833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdlg\" (UniqueName: \"kubernetes.io/projected/763f05fe-0d04-49e8-896c-c5b095c221e5-kube-api-access-ksdlg\") pod \"iptables-alerter-9486k\" (UID: \"763f05fe-0d04-49e8-896c-c5b095c221e5\") " pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.754949 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.754891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6sx\" (UniqueName: \"kubernetes.io/projected/65425295-4d07-4177-a298-158626a08624-kube-api-access-7c6sx\") pod \"aws-ebs-csi-driver-node-v79mr\" (UID: \"65425295-4d07-4177-a298-158626a08624\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.844987 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.844956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:17:48.852291 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.852263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fb11b7_fbab_4120_87a5_e4dce9efc13c.slice/crio-8e3262091d33bc7ac8e5bae56c63fc43e14e0e7ba2502373c971f639a035a7a5 WatchSource:0}: Error finding container 8e3262091d33bc7ac8e5bae56c63fc43e14e0e7ba2502373c971f639a035a7a5: Status 404 returned error can't find the container with id 8e3262091d33bc7ac8e5bae56c63fc43e14e0e7ba2502373c971f639a035a7a5
Jul 27 20:17:48.871537 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.871502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5qpn"
Jul 27 20:17:48.878598 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.878573    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7f7396_c4d5_4a5d_86c9_7a2e013fb365.slice/crio-e9510f0d2e3a4b6a6cab8be79b2c400649e1e714c2689cbdcba80234fb339df4 WatchSource:0}: Error finding container e9510f0d2e3a4b6a6cab8be79b2c400649e1e714c2689cbdcba80234fb339df4: Status 404 returned error can't find the container with id e9510f0d2e3a4b6a6cab8be79b2c400649e1e714c2689cbdcba80234fb339df4
Jul 27 20:17:48.890314 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.890290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:17:48.897161 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.897137    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e16b3df_cbd5_4241_99b8_e199148b68e0.slice/crio-b3bef8ed6d5278ea72d3bfd1f0d6669cc2d46a22940ea488885b31418af358a0 WatchSource:0}: Error finding container b3bef8ed6d5278ea72d3bfd1f0d6669cc2d46a22940ea488885b31418af358a0: Status 404 returned error can't find the container with id b3bef8ed6d5278ea72d3bfd1f0d6669cc2d46a22940ea488885b31418af358a0
Jul 27 20:17:48.900234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.900215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p9v6l"
Jul 27 20:17:48.908401 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.908374    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0b6962_d577_4475_b9c6_c098e90435b7.slice/crio-73b66d2db427474b17f9040df25728ad288eb4b666e02014dc3c8f9cbecc5500 WatchSource:0}: Error finding container 73b66d2db427474b17f9040df25728ad288eb4b666e02014dc3c8f9cbecc5500: Status 404 returned error can't find the container with id 73b66d2db427474b17f9040df25728ad288eb4b666e02014dc3c8f9cbecc5500
Jul 27 20:17:48.914145 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.914124    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8gdvn"
Jul 27 20:17:48.919888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.919867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dq6h9"
Jul 27 20:17:48.921237 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.921215    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d905ce2_dab1_4209_afa0_8107f39f9124.slice/crio-943de261e4eb84b17858b8612d61f7d0db1cefed29673e18baed82c5946c4281 WatchSource:0}: Error finding container 943de261e4eb84b17858b8612d61f7d0db1cefed29673e18baed82c5946c4281: Status 404 returned error can't find the container with id 943de261e4eb84b17858b8612d61f7d0db1cefed29673e18baed82c5946c4281
Jul 27 20:17:48.927453 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.927429    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2a7e81_83ab_41cc_b66d_c1ae1592ad14.slice/crio-bf61db4a7099b3f44546c83aeb7d37be649afe5ce296bde2c0da138025227bcc WatchSource:0}: Error finding container bf61db4a7099b3f44546c83aeb7d37be649afe5ce296bde2c0da138025227bcc: Status 404 returned error can't find the container with id bf61db4a7099b3f44546c83aeb7d37be649afe5ce296bde2c0da138025227bcc
Jul 27 20:17:48.943574 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.943533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-crkxs"
Jul 27 20:17:48.951184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.951162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr"
Jul 27 20:17:48.951448 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.951426    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba724cb_8bb8_4f73_be21_ab451f82a6ff.slice/crio-66f0beeeacbfd1329f873d9398991bc8fabcbb5cf7a3784995e9c019da889daf WatchSource:0}: Error finding container 66f0beeeacbfd1329f873d9398991bc8fabcbb5cf7a3784995e9c019da889daf: Status 404 returned error can't find the container with id 66f0beeeacbfd1329f873d9398991bc8fabcbb5cf7a3784995e9c019da889daf
Jul 27 20:17:48.956635 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:48.956612    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9486k"
Jul 27 20:17:48.959159 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.959133    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65425295_4d07_4177_a298_158626a08624.slice/crio-d8dfe956aa38e3abeac58bb8575211b4ef5d4ad353ee60f65ce9522b9532c9df WatchSource:0}: Error finding container d8dfe956aa38e3abeac58bb8575211b4ef5d4ad353ee60f65ce9522b9532c9df: Status 404 returned error can't find the container with id d8dfe956aa38e3abeac58bb8575211b4ef5d4ad353ee60f65ce9522b9532c9df
Jul 27 20:17:48.964193 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:17:48.964170    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763f05fe_0d04_49e8_896c_c5b095c221e5.slice/crio-e3118f8a7ab0cd7859e40b536b421cd405a999a733008aa25ceb29f6403da839 WatchSource:0}: Error finding container e3118f8a7ab0cd7859e40b536b421cd405a999a733008aa25ceb29f6403da839: Status 404 returned error can't find the container with id e3118f8a7ab0cd7859e40b536b421cd405a999a733008aa25ceb29f6403da839
Jul 27 20:17:49.148002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.147964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:49.148002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.148018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:49.148246 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.148145    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:49.148246 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.148219    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:50.148198104 +0000 UTC m=+2.997964933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:49.148246 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.148155    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:49.148393 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.148282    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:50.148264942 +0000 UTC m=+2.998031757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:49.249127 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.249039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:49.249303 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.249208    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:17:49.249303 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.249231    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:17:49.249303 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.249244    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:49.249303 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.249303    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:50.249284924 +0000 UTC m=+3.099051737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:49.571968 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.571839    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:12:48 +0000 UTC" deadline="2028-03-09 20:52:54.768759894 +0000 UTC"
Jul 27 20:17:49.571968 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.571915    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14184h35m5.196850397s"
Jul 27 20:17:49.588689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.588656    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:49.720758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.720722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:49.720940 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.720880    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:49.721273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.720681    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:49.721381 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:49.721358    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:49.754382 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.754251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9486k" event={"ID":"763f05fe-0d04-49e8-896c-c5b095c221e5","Type":"ContainerStarted","Data":"e3118f8a7ab0cd7859e40b536b421cd405a999a733008aa25ceb29f6403da839"}
Jul 27 20:17:49.769310 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.769267    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crkxs" event={"ID":"dba724cb-8bb8-4f73-be21-ab451f82a6ff","Type":"ContainerStarted","Data":"66f0beeeacbfd1329f873d9398991bc8fabcbb5cf7a3784995e9c019da889daf"}
Jul 27 20:17:49.781312 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.781179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerStarted","Data":"bf61db4a7099b3f44546c83aeb7d37be649afe5ce296bde2c0da138025227bcc"}
Jul 27 20:17:49.785575 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.785484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gdvn" event={"ID":"7d905ce2-dab1-4209-afa0-8107f39f9124","Type":"ContainerStarted","Data":"943de261e4eb84b17858b8612d61f7d0db1cefed29673e18baed82c5946c4281"}
Jul 27 20:17:49.805133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.805007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"b3bef8ed6d5278ea72d3bfd1f0d6669cc2d46a22940ea488885b31418af358a0"}
Jul 27 20:17:49.813028 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.812997    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:49.828331 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.828181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z5hqg" event={"ID":"61fb11b7-fbab-4120-87a5-e4dce9efc13c","Type":"ContainerStarted","Data":"8e3262091d33bc7ac8e5bae56c63fc43e14e0e7ba2502373c971f639a035a7a5"}
Jul 27 20:17:49.861158 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.861119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal" event={"ID":"198dc54988f9644415b48f7e98d8dd24","Type":"ContainerStarted","Data":"c714df7c03e330ed61257045ee75883802299f07d38ed38389f95ba0a1b4ebbc"}
Jul 27 20:17:49.870038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.869997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr" event={"ID":"65425295-4d07-4177-a298-158626a08624","Type":"ContainerStarted","Data":"d8dfe956aa38e3abeac58bb8575211b4ef5d4ad353ee60f65ce9522b9532c9df"}
Jul 27 20:17:49.889054 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.888932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9v6l" event={"ID":"1a0b6962-d577-4475-b9c6-c098e90435b7","Type":"ContainerStarted","Data":"73b66d2db427474b17f9040df25728ad288eb4b666e02014dc3c8f9cbecc5500"}
Jul 27 20:17:49.912706 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:49.912655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5qpn" event={"ID":"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365","Type":"ContainerStarted","Data":"e9510f0d2e3a4b6a6cab8be79b2c400649e1e714c2689cbdcba80234fb339df4"}
Jul 27 20:17:50.158188 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.158100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:50.158188 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.158168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:50.158417 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.158343    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:50.158417 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.158410    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:52.158390274 +0000 UTC m=+5.008157082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:50.160622 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.158789    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:50.160622 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.158869    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:52.158853823 +0000 UTC m=+5.008620637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:50.258559 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.258519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:50.258764 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.258745    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:17:50.258906 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.258771    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:17:50.258906 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.258784    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:50.258906 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.258857    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:52.258838599 +0000 UTC m=+5.108605411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:50.291277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.291244    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:17:50.572840 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.572734    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:12:48 +0000 UTC" deadline="2028-04-03 06:27:31.209787978 +0000 UTC"
Jul 27 20:17:50.572840 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.572771    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14770h9m40.637021782s"
Jul 27 20:17:50.721016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:50.720982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:50.721203 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:50.721120    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:51.721190 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:51.720687    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:51.721190 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:51.720715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:51.721190 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:51.720844    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:51.721190 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:51.720921    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:52.173738 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:52.173700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:52.173955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:52.173754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:52.173955 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.173898    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:52.173955 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.173898    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:52.174115 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.173968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:56.173947751 +0000 UTC m=+9.023714572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:52.174115 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.173988    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:56.173978616 +0000 UTC m=+9.023745429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:52.274403 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:52.274365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:52.274593 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.274546    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:17:52.274593 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.274577    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:17:52.274593 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.274590    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:52.274742 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.274655    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:17:56.274633562 +0000 UTC m=+9.124400371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:52.721049 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:52.720539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:52.721049 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:52.720679    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:53.720915 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:53.720878    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:53.721084 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:53.721031    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:53.721228 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:53.721207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:53.721343 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:53.721323    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:54.720777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:54.720744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:54.720993 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:54.720880    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:55.724213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:55.724129    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:55.724655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:55.724129    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:55.724655 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:55.724264    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:55.724655 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:55.724356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:56.211278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:56.211332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.211420    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.211484    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:04.211465856 +0000 UTC m=+17.061232668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.211420    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:56.211891 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.211540    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:04.2115286 +0000 UTC m=+17.061295421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:17:56.312153 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:56.312114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:56.312343 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.312296    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:17:56.312343 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.312318    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:17:56.312343 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.312330    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:56.312506 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.312394    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:04.312373682 +0000 UTC m=+17.162140510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:17:56.720355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:56.720317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:56.720540 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:56.720449    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:57.724629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:57.724589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:57.725118 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:57.724727    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:57.725118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:57.724781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:57.725118 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:57.724906    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:17:58.720169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:58.720134    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:17:58.720349 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:58.720260    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:17:59.721077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:59.721033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:17:59.721077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:17:59.721054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:17:59.721578 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:59.721164    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:17:59.721578 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:17:59.721301    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:00.720950 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:00.720872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:00.721107 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:00.721032    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:01.720457 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:01.720425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:01.720657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:01.720435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:01.720657 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:01.720538    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:01.720657 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:01.720638    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:02.720561 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:02.720523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:02.720988 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:02.720659    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:03.720535 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:03.720499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:03.720731 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:03.720619    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:03.720731 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:03.720680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:03.721143 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:03.720792    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:04.272755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:04.272653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:04.272755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:04.272703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:04.273012 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.272825    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:18:04.273012 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.272850    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:18:04.273012 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.272898    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:20.272878102 +0000 UTC m=+33.122644921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:18:04.273012 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.272919    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:20.272907914 +0000 UTC m=+33.122674722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:18:04.373449 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:04.373411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:04.373648 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.373574    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:18:04.373648 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.373597    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:18:04.373648 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.373608    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:18:04.373793 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.373671    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:20.373652361 +0000 UTC m=+33.223419183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:18:04.720172 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:04.720095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:04.720395 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:04.720223    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:05.720502 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:05.720461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:05.720990 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:05.720586    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:05.720990 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:05.720652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:05.720990 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:05.720784    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:06.720959 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:06.720922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:06.721347 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:06.721039    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:07.720694 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:07.720661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:07.720891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:07.720781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:07.720957 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:07.720777    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:07.720957 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:07.720919    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:08.720457 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.720175    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:08.720595 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:08.720554    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:08.957558 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.957349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal" event={"ID":"f84ce0d9219c2a58af44c132cfd94a88","Type":"ContainerStarted","Data":"b226dc58263ddc948e0f50032269856ad938be726afa34666ee72567c5873e92"}
Jul 27 20:18:08.958782 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.958757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-crkxs" event={"ID":"dba724cb-8bb8-4f73-be21-ab451f82a6ff","Type":"ContainerStarted","Data":"1ed84f0948a5a65aa1659737f1d927de7e45eb4e771b90c6af33499781e60332"}
Jul 27 20:18:08.960405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.960376    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="4e13f2e088b6e652012d20a7d04f76313c728b455ca2b6cf60f59f9d130eac9f" exitCode=0
Jul 27 20:18:08.960518 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.960407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"4e13f2e088b6e652012d20a7d04f76313c728b455ca2b6cf60f59f9d130eac9f"}
Jul 27 20:18:08.962127 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.962100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8gdvn" event={"ID":"7d905ce2-dab1-4209-afa0-8107f39f9124","Type":"ContainerStarted","Data":"9bbe3f06f5f03db0f7a1bd8596c29618480316ad26110e1824aed53d863f3222"}
Jul 27 20:18:08.963695 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.963673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"365aad2f2849b15fb3bdb9eafeea4feaac628edbd505fe32bdc5e51a88eca91a"}
Jul 27 20:18:08.963774 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.963733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"617163a53058d33be3e62de21da831107dde4b44387c0d8539fe7d339062ba8f"}
Jul 27 20:18:08.965135 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.965109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z5hqg" event={"ID":"61fb11b7-fbab-4120-87a5-e4dce9efc13c","Type":"ContainerStarted","Data":"3511172e21836de8130782b310c6f7d4d380619218dafd6c5c3b736010da1636"}
Jul 27 20:18:08.966527 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.966506    2560 generic.go:356] "Generic (PLEG): container finished" podID="198dc54988f9644415b48f7e98d8dd24" containerID="c224554809f83e85cd01bc9cd75c363910a07b5acdf66c0166b34bee37134745" exitCode=0
Jul 27 20:18:08.966609 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.966570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal" event={"ID":"198dc54988f9644415b48f7e98d8dd24","Type":"ContainerDied","Data":"c224554809f83e85cd01bc9cd75c363910a07b5acdf66c0166b34bee37134745"}
Jul 27 20:18:08.967906 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.967888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr" event={"ID":"65425295-4d07-4177-a298-158626a08624","Type":"ContainerStarted","Data":"1949ebfa02301abf660f18732f89d884eca9a0cb13607e3c301977237343f03f"}
Jul 27 20:18:08.969577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.969560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p9v6l" event={"ID":"1a0b6962-d577-4475-b9c6-c098e90435b7","Type":"ContainerStarted","Data":"13d174578ba416c411c5007bfae8c752d264134cb7787de97bc4261095b943fe"}
Jul 27 20:18:08.970966 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.970925    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-74.ec2.internal" podStartSLOduration=20.970911189 podStartE2EDuration="20.970911189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:08.970196691 +0000 UTC m=+21.819963521" watchObservedRunningTime="2026-07-27 20:18:08.970911189 +0000 UTC m=+21.820678017"
Jul 27 20:18:08.970966 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:08.970962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5qpn" event={"ID":"5e7f7396-c4d5-4a5d-86c9-7a2e013fb365","Type":"ContainerStarted","Data":"c27b24653d826ec4c5161c82d129bf1b99d6262c6515dd47377300801906c440"}
Jul 27 20:18:09.005484 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.005422    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z5hqg" podStartSLOduration=10.419542756 podStartE2EDuration="22.005402791s" totalImagesPullingTime="11.585860035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:08.989081017 +0000 UTC m=+21.838847846" watchObservedRunningTime="2026-07-27 20:18:09.005402791 +0000 UTC m=+21.855169620"
Jul 27 20:18:09.005609 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.005536    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p9v6l" podStartSLOduration=2.512065174 podStartE2EDuration="22.00552529s" totalImagesPullingTime="19.493460116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:09.005023878 +0000 UTC m=+21.854790709" watchObservedRunningTime="2026-07-27 20:18:09.00552529 +0000 UTC m=+21.855292121"
Jul 27 20:18:09.018651 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.018608    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-crkxs" podStartSLOduration=2.592635863 podStartE2EDuration="22.018593281s" totalImagesPullingTime="19.425957418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:09.018295618 +0000 UTC m=+21.868062447" watchObservedRunningTime="2026-07-27 20:18:09.018593281 +0000 UTC m=+21.868360110"
Jul 27 20:18:09.031921 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.031860    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8gdvn" podStartSLOduration=2.55279242 podStartE2EDuration="22.031845608s" totalImagesPullingTime="19.479053188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:09.031805013 +0000 UTC m=+21.881571863" watchObservedRunningTime="2026-07-27 20:18:09.031845608 +0000 UTC m=+21.881612437"
Jul 27 20:18:09.720627 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.720537    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:09.720864 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.720544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:09.720864 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:09.720679    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:09.720864 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:09.720766    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:09.724749 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.724727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:18:09.724908 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.724867    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:18:09.739859 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.739766    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j5qpn" podStartSLOduration=3.206709979 podStartE2EDuration="22.739744085s" totalImagesPullingTime="19.533034106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:09.072582082 +0000 UTC m=+21.922348934" watchObservedRunningTime="2026-07-27 20:18:09.739744085 +0000 UTC m=+22.589510919"
Jul 27 20:18:09.977300 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.977219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"c2ed9851712288ab7a5a280bad8c9a381c74d74e798e5eb9e50ecb141ff21814"}
Jul 27 20:18:09.977980 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.977311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"eed6e4cfc0ac43d1cf11d41ad22b37e5ecf4f54d79e7e58fdd1e7e397e95319d"}
Jul 27 20:18:09.977980 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.977357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"8f577aeac7b7572adbb1f16f674414370654900aa7709303b14b76950e34fd91"}
Jul 27 20:18:09.977980 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.977402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"641f1aa57b1830419067b182ae66867a99f5ca37816db3940160522776527ce6"}
Jul 27 20:18:09.978973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.978939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9486k" event={"ID":"763f05fe-0d04-49e8-896c-c5b095c221e5","Type":"ContainerStarted","Data":"ecd6b1a7924ca2b27245f3715857ac594ffec39cedeec9ab494f45f9d976e00a"}
Jul 27 20:18:09.980042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.980017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:18:09.980506 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:09.980477    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z5hqg"
Jul 27 20:18:10.008797 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.008743    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9486k" podStartSLOduration=3.660172395 podStartE2EDuration="23.0087236s" totalImagesPullingTime="19.348551205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:10.008196175 +0000 UTC m=+22.857963001" watchObservedRunningTime="2026-07-27 20:18:10.0087236 +0000 UTC m=+22.858490430"
Jul 27 20:18:10.288477 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.288239    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:18:10.604648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.601870    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:18:10.288468508Z","UUID":"bfb56df5-e721-42d0-810c-11a41f5465f0","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:18:10.604962 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.604665    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:18:10.604962 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.604746    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:18:10.720556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.720525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:10.720767 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:10.720659    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:10.982700 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.982616    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal" event={"ID":"198dc54988f9644415b48f7e98d8dd24","Type":"ContainerStarted","Data":"f145f7b804eafe0b9e6b7fd29a56cf34f0466c5cf0ecbb80d294d7b39a401ad5"}
Jul 27 20:18:10.984875 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.984840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr" event={"ID":"65425295-4d07-4177-a298-158626a08624","Type":"ContainerStarted","Data":"2219008ce0627fb9979c37d2c12f0f0621c72186b5dba86136a904ec6362816f"}
Jul 27 20:18:10.998552 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:10.998490    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-74.ec2.internal" podStartSLOduration=22.99846972 podStartE2EDuration="22.99846972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:10.997725377 +0000 UTC m=+23.847492209" watchObservedRunningTime="2026-07-27 20:18:10.99846972 +0000 UTC m=+23.848236551"
Jul 27 20:18:11.721166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:11.721046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:11.721166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:11.721065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:11.721376 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:11.721190    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:11.721376 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:11.721336    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:11.977903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:11.977834    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crkxs_dba724cb-8bb8-4f73-be21-ab451f82a6ff/dns-node-resolver/0.log"
Jul 27 20:18:11.989239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:11.989211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"f36a947d08392428dd6746739d82bc4a4cba044189adde76962733c85bb3edf7"}
Jul 27 20:18:11.991011 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:11.990980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr" event={"ID":"65425295-4d07-4177-a298-158626a08624","Type":"ContainerStarted","Data":"386bf1e1b88d2e54918893d7726616966704f9f76225260469c0f9554cc42b18"}
Jul 27 20:18:12.010509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:12.010458    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v79mr" podStartSLOduration=3.487811738 podStartE2EDuration="25.01044423s" totalImagesPullingTime="21.522632492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:12.010278208 +0000 UTC m=+24.860045048" watchObservedRunningTime="2026-07-27 20:18:12.01044423 +0000 UTC m=+24.860211059"
Jul 27 20:18:12.720304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:12.720261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:12.720495 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:12.720405    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:12.751629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:12.751605    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gdvn_7d905ce2-dab1-4209-afa0-8107f39f9124/node-ca/0.log"
Jul 27 20:18:13.721405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:13.721193    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:13.721405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:13.721194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:13.721405 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:13.721323    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:13.722474 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:13.721435    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:13.997213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:13.997118    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="3fd0bf6c834066d380b6c852eeeea3c50e58e91b2dbb73bc09e3cda50ae6b951" exitCode=0
Jul 27 20:18:13.997213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:13.997199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"3fd0bf6c834066d380b6c852eeeea3c50e58e91b2dbb73bc09e3cda50ae6b951"}
Jul 27 20:18:14.000890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.000860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" event={"ID":"4e16b3df-cbd5-4241-99b8-e199148b68e0","Type":"ContainerStarted","Data":"0c5e1c8dc04e0cfd6a07925981f40e85a0f0762bd3aee5dcab935da7548c69dd"}
Jul 27 20:18:14.001237 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.001217    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:14.001301 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.001289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:14.016359 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.016331    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:14.016504 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.016443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:14.057803 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.057752    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" podStartSLOduration=7.347678945 podStartE2EDuration="27.057735795s" totalImagesPullingTime="19.71005685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:14.056209917 +0000 UTC m=+26.905976746" watchObservedRunningTime="2026-07-27 20:18:14.057735795 +0000 UTC m=+26.907502625"
Jul 27 20:18:14.720964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:14.720924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:14.721215 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:14.721055    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:15.002845 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.002760    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 20:18:15.721384 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.721354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:15.721571 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.721401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:15.721571 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:15.721528    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:15.721685 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:15.721659    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:15.895196 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.895163    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npmtt"]
Jul 27 20:18:15.898540 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.898511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7llv"]
Jul 27 20:18:15.898674 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.898642    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:15.898761 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:15.898741    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:15.899270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:15.899250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvsbh"]
Jul 27 20:18:16.006610 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:16.006521    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="112fe15644e009911a4fc248177c48d1bd5a330852239dc5ec2c5ce21bb1e66d" exitCode=0
Jul 27 20:18:16.006610 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:16.006600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:16.007323 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:16.006627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"112fe15644e009911a4fc248177c48d1bd5a330852239dc5ec2c5ce21bb1e66d"}
Jul 27 20:18:16.007323 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:16.006868    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:16.007323 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:16.006926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:16.007323 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:16.007006    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:16.007323 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:16.007099    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 20:18:17.011013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:17.010824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerStarted","Data":"6ab7d379de9f318e6e3bb946fcf04565c1199b98b66411f9d3be10e83f17ed00"}
Jul 27 20:18:17.721657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:17.721626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:17.721800 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:17.721699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:17.721800 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:17.721721    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:17.721946 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:17.721830    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:17.721946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:17.721914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:17.722039 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:17.722018    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:18.016037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.016000    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="6ab7d379de9f318e6e3bb946fcf04565c1199b98b66411f9d3be10e83f17ed00" exitCode=0
Jul 27 20:18:18.016462 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.016067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"6ab7d379de9f318e6e3bb946fcf04565c1199b98b66411f9d3be10e83f17ed00"}
Jul 27 20:18:18.441239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.441197    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:18.441434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.441407    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 20:18:18.460434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.460393    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" podUID="4e16b3df-cbd5-4241-99b8-e199148b68e0" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 27 20:18:18.469072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:18.469034    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-nk775" podUID="4e16b3df-cbd5-4241-99b8-e199148b68e0" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 27 20:18:19.720514 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:19.720483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:19.721191 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:19.720486    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:19.721191 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:19.720622    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:19.721191 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:19.720713    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:19.721191 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:19.720497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:19.721191 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:19.720832    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:20.296602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:20.296562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:20.296783 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:20.296701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:20.296783 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.296726    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:18:20.296912 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.296799    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:18:20.296912 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.296826    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret podName:3c95815a-2734-4e41-82bf-bd4d0f5883f2 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:52.296787396 +0000 UTC m=+65.146554208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret") pod "global-pull-secret-syncer-npmtt" (UID: "3c95815a-2734-4e41-82bf-bd4d0f5883f2") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:18:20.296912 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.296864    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs podName:1da0167a-bfd5-4791-a03c-52fc3795d410 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:52.29685217 +0000 UTC m=+65.146618978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs") pod "network-metrics-daemon-gvsbh" (UID: "1da0167a-bfd5-4791-a03c-52fc3795d410") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:18:20.397583 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:20.397544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:20.397842 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.397721    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:18:20.397842 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.397742    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:18:20.397842 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.397756    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w26vg for pod openshift-network-diagnostics/network-check-target-t7llv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:18:20.397842 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:20.397834    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg podName:f727887c-fe02-4346-8277-7a51f96728f6 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:52.397804091 +0000 UTC m=+65.247570899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w26vg" (UniqueName: "kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg") pod "network-check-target-t7llv" (UID: "f727887c-fe02-4346-8277-7a51f96728f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:18:21.721159 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:21.721123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:21.721685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:21.721126    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:21.721685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:21.721260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:21.721685 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:21.721256    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:21.721685 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:21.721382    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:21.721685 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:21.721440    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:23.029790 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:23.029759    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="d3ab8f63abddf8bb78dd4c67188b8ca4967f222223a4d7ccd4658b7e5ddaf898" exitCode=0
Jul 27 20:18:23.030169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:23.029819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"d3ab8f63abddf8bb78dd4c67188b8ca4967f222223a4d7ccd4658b7e5ddaf898"}
Jul 27 20:18:23.720434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:23.720389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:23.720434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:23.720418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:23.720689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:23.720391    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:23.720689 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:23.720495    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:23.720689 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:23.720575    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:23.720689 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:23.720635    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:24.034495 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:24.034403    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f2a7e81-83ab-41cc-b66d-c1ae1592ad14" containerID="ebdd68d1d5a54428e4d704e9772e67fbad5dac16d6435c677757fdc33d7fda3f" exitCode=0
Jul 27 20:18:24.034495 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:24.034444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerDied","Data":"ebdd68d1d5a54428e4d704e9772e67fbad5dac16d6435c677757fdc33d7fda3f"}
Jul 27 20:18:25.040174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:25.039986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" event={"ID":"3f2a7e81-83ab-41cc-b66d-c1ae1592ad14","Type":"ContainerStarted","Data":"0e65ee99f3c44f3d343c94f94f60cdd3666eee80685e6e8fa2f6856f1a061b27"}
Jul 27 20:18:25.061685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:25.061644    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dq6h9" podStartSLOduration=7.620981277 podStartE2EDuration="38.061629011s" totalImagesPullingTime="30.440647734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:25.061472835 +0000 UTC m=+37.911239667" watchObservedRunningTime="2026-07-27 20:18:25.061629011 +0000 UTC m=+37.911395840"
Jul 27 20:18:25.720657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:25.720568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:25.720790 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:25.720572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:25.720790 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:25.720684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:25.720790 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:25.720572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:25.720790 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:25.720758    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:25.720952 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:25.720839    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:27.721216 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:27.721175    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:27.721699 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:27.721275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:27.721699 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:27.721292    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:27.721699 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:27.721313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:27.721699 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:27.721401    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:27.721699 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:27.721482    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:29.720399 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:29.720360    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:29.720784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:29.720371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:29.720784 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:29.720479    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:29.720784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:29.720573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:29.720784 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:29.720574    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:29.720784 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:29.720669    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:31.721019 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:31.720983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:31.721475 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:31.720987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:31.721475 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:31.721094    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:31.721475 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:31.721194    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:31.721475 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:31.720986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:31.721475 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:31.721286    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:33.720218 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:33.720180    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:33.720598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:33.720181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:33.720598 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:33.720287    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:33.720598 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:33.720375    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:33.720598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:33.720418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:33.720598 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:33.720479    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:35.720555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:35.720518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:35.720555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:35.720544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:35.721074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:35.720518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:35.721074 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:35.720637    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npmtt" podUID="3c95815a-2734-4e41-82bf-bd4d0f5883f2"
Jul 27 20:18:35.721074 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:35.720698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvsbh" podUID="1da0167a-bfd5-4791-a03c-52fc3795d410"
Jul 27 20:18:35.721074 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:35.720796    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7llv" podUID="f727887c-fe02-4346-8277-7a51f96728f6"
Jul 27 20:18:36.460233 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.460207    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-74.ec2.internal" event="NodeReady"
Jul 27 20:18:36.460371 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.460334    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:18:36.505036 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.505003    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-552n5"]
Jul 27 20:18:36.513192 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.513162    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xnms9"]
Jul 27 20:18:36.513334 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.513280    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.515677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.515654    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4zfst\""
Jul 27 20:18:36.516016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.515991    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:18:36.517183 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.517091    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:18:36.524673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.524650    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-552n5"]
Jul 27 20:18:36.524777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.524694    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnms9"]
Jul 27 20:18:36.524834 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.524784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.527528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.527510    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qtb5t\""
Jul 27 20:18:36.527528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.527510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:18:36.527689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.527558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:18:36.528099 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.527893    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:18:36.528794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.528778    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fwgnd"]
Jul 27 20:18:36.542069 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.542042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.543617 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.543590    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fwgnd"]
Jul 27 20:18:36.544653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.544634    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:18:36.544961 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.544944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:18:36.545447 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.545425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:18:36.545528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.545454    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdwph\""
Jul 27 20:18:36.545528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.545499    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:18:36.614385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.614343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1f8bc-723b-4b13-a86c-93c3f0037210-config-volume\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.614385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.614393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c1f8bc-723b-4b13-a86c-93c3f0037210-metrics-tls\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.614600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.614410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c1f8bc-723b-4b13-a86c-93c3f0037210-tmp-dir\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.614600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.614430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf66l\" (UniqueName: \"kubernetes.io/projected/f5c1f8bc-723b-4b13-a86c-93c3f0037210-kube-api-access-xf66l\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.714797 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m885d\" (UniqueName: \"kubernetes.io/projected/32bab317-fce0-4e9c-9f72-af34be8b63e0-kube-api-access-m885d\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.714797 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012af03a-194e-44ab-9e01-54e6427580d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1f8bc-723b-4b13-a86c-93c3f0037210-config-volume\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012af03a-194e-44ab-9e01-54e6427580d2-data-volume\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c1f8bc-723b-4b13-a86c-93c3f0037210-metrics-tls\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012af03a-194e-44ab-9e01-54e6427580d2-crio-socket\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c1f8bc-723b-4b13-a86c-93c3f0037210-tmp-dir\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqss\" (UniqueName: \"kubernetes.io/projected/012af03a-194e-44ab-9e01-54e6427580d2-kube-api-access-4qqss\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.714976 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf66l\" (UniqueName: \"kubernetes.io/projected/f5c1f8bc-723b-4b13-a86c-93c3f0037210-kube-api-access-xf66l\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.715246 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.714999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012af03a-194e-44ab-9e01-54e6427580d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.715246 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.715021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32bab317-fce0-4e9c-9f72-af34be8b63e0-cert\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.715354 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.715272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c1f8bc-723b-4b13-a86c-93c3f0037210-tmp-dir\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.715397 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.715387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c1f8bc-723b-4b13-a86c-93c3f0037210-config-volume\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.719110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.719087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c1f8bc-723b-4b13-a86c-93c3f0037210-metrics-tls\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.722736 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.722708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf66l\" (UniqueName: \"kubernetes.io/projected/f5c1f8bc-723b-4b13-a86c-93c3f0037210-kube-api-access-xf66l\") pod \"dns-default-552n5\" (UID: \"f5c1f8bc-723b-4b13-a86c-93c3f0037210\") " pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.815767 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012af03a-194e-44ab-9e01-54e6427580d2-crio-socket\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.815767 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqss\" (UniqueName: \"kubernetes.io/projected/012af03a-194e-44ab-9e01-54e6427580d2-kube-api-access-4qqss\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012af03a-194e-44ab-9e01-54e6427580d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32bab317-fce0-4e9c-9f72-af34be8b63e0-cert\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m885d\" (UniqueName: \"kubernetes.io/projected/32bab317-fce0-4e9c-9f72-af34be8b63e0-kube-api-access-m885d\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012af03a-194e-44ab-9e01-54e6427580d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012af03a-194e-44ab-9e01-54e6427580d2-data-volume\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.815976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/012af03a-194e-44ab-9e01-54e6427580d2-crio-socket\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816325 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.816283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/012af03a-194e-44ab-9e01-54e6427580d2-data-volume\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.816425 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.816408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/012af03a-194e-44ab-9e01-54e6427580d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.818097 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.818078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/012af03a-194e-44ab-9e01-54e6427580d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.818271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.818256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32bab317-fce0-4e9c-9f72-af34be8b63e0-cert\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.822870 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.822803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:36.824280 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.824261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m885d\" (UniqueName: \"kubernetes.io/projected/32bab317-fce0-4e9c-9f72-af34be8b63e0-kube-api-access-m885d\") pod \"ingress-canary-xnms9\" (UID: \"32bab317-fce0-4e9c-9f72-af34be8b63e0\") " pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.824358 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.824321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqss\" (UniqueName: \"kubernetes.io/projected/012af03a-194e-44ab-9e01-54e6427580d2-kube-api-access-4qqss\") pod \"insights-runtime-extractor-fwgnd\" (UID: \"012af03a-194e-44ab-9e01-54e6427580d2\") " pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:36.833254 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.833227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xnms9"
Jul 27 20:18:36.851323 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:36.851303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fwgnd"
Jul 27 20:18:37.028426 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.028394    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-552n5"]
Jul 27 20:18:37.031284 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:37.031255    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c1f8bc_723b_4b13_a86c_93c3f0037210.slice/crio-65eb3d3c52c3ccfa4bb82e6377afec276dcaaf559025c3d3afb8a775d2e50041 WatchSource:0}: Error finding container 65eb3d3c52c3ccfa4bb82e6377afec276dcaaf559025c3d3afb8a775d2e50041: Status 404 returned error can't find the container with id 65eb3d3c52c3ccfa4bb82e6377afec276dcaaf559025c3d3afb8a775d2e50041
Jul 27 20:18:37.038371 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.038340    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fwgnd"]
Jul 27 20:18:37.039091 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.039069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xnms9"]
Jul 27 20:18:37.042142 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:37.042122    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bab317_fce0_4e9c_9f72_af34be8b63e0.slice/crio-05c8d5c41528378e3d2481900e2e4d325c77b9dba047489a8d89eec016f06565 WatchSource:0}: Error finding container 05c8d5c41528378e3d2481900e2e4d325c77b9dba047489a8d89eec016f06565: Status 404 returned error can't find the container with id 05c8d5c41528378e3d2481900e2e4d325c77b9dba047489a8d89eec016f06565
Jul 27 20:18:37.042847 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:37.042804    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012af03a_194e_44ab_9e01_54e6427580d2.slice/crio-1cf6a32cad5a0a9880ee711a02691071a9fa46d426d00302b97d7b31895c1ee5 WatchSource:0}: Error finding container 1cf6a32cad5a0a9880ee711a02691071a9fa46d426d00302b97d7b31895c1ee5: Status 404 returned error can't find the container with id 1cf6a32cad5a0a9880ee711a02691071a9fa46d426d00302b97d7b31895c1ee5
Jul 27 20:18:37.067336 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.067304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwgnd" event={"ID":"012af03a-194e-44ab-9e01-54e6427580d2","Type":"ContainerStarted","Data":"1cf6a32cad5a0a9880ee711a02691071a9fa46d426d00302b97d7b31895c1ee5"}
Jul 27 20:18:37.068353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.068330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnms9" event={"ID":"32bab317-fce0-4e9c-9f72-af34be8b63e0","Type":"ContainerStarted","Data":"05c8d5c41528378e3d2481900e2e4d325c77b9dba047489a8d89eec016f06565"}
Jul 27 20:18:37.069271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.069244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-552n5" event={"ID":"f5c1f8bc-723b-4b13-a86c-93c3f0037210","Type":"ContainerStarted","Data":"65eb3d3c52c3ccfa4bb82e6377afec276dcaaf559025c3d3afb8a775d2e50041"}
Jul 27 20:18:37.726566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.725072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:37.726566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.725616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:37.726566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.726194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:37.731013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.730986    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:18:37.733413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.732253    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xjfvv\""
Jul 27 20:18:37.733413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.732486    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:18:37.733413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.732694    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:18:37.733413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.732952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvk7\""
Jul 27 20:18:37.733413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:37.733189    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:18:40.316073 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.316041    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"]
Jul 27 20:18:40.332220 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.332184    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"]
Jul 27 20:18:40.332359 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.332283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:40.335206 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.335182    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 20:18:40.335502 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.335489    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kbpkj\""
Jul 27 20:18:40.343929 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.343898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-frw9p\" (UID: \"1a74d76a-4fff-49b3-96f4-f8605dbace98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:40.445237 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.445199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-frw9p\" (UID: \"1a74d76a-4fff-49b3-96f4-f8605dbace98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:40.445407 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:40.445366    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:18:40.445575 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:40.445449    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates podName:1a74d76a-4fff-49b3-96f4-f8605dbace98 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:40.94542397 +0000 UTC m=+53.795190779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-frw9p" (UID: "1a74d76a-4fff-49b3-96f4-f8605dbace98") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:18:40.948121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.948079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-frw9p\" (UID: \"1a74d76a-4fff-49b3-96f4-f8605dbace98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:40.950695 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:40.950658    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a74d76a-4fff-49b3-96f4-f8605dbace98-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-frw9p\" (UID: \"1a74d76a-4fff-49b3-96f4-f8605dbace98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:41.242271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:41.242182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:43.020775 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.020691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"]
Jul 27 20:18:43.025857 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:43.025795    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a74d76a_4fff_49b3_96f4_f8605dbace98.slice/crio-118396c1612f051916de72e8c8bc0bfe1066b8ab454858ae417a6c087030101d WatchSource:0}: Error finding container 118396c1612f051916de72e8c8bc0bfe1066b8ab454858ae417a6c087030101d: Status 404 returned error can't find the container with id 118396c1612f051916de72e8c8bc0bfe1066b8ab454858ae417a6c087030101d
Jul 27 20:18:43.088887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.088849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwgnd" event={"ID":"012af03a-194e-44ab-9e01-54e6427580d2","Type":"ContainerStarted","Data":"ab88ffc240448564f8f04d64c3023db0060991785e19fabecab203848df3304f"}
Jul 27 20:18:43.090556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.090518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xnms9" event={"ID":"32bab317-fce0-4e9c-9f72-af34be8b63e0","Type":"ContainerStarted","Data":"0b04951ceca9c15e92151b8477d362280e38288f03019c8b5d92851dbe7c5cea"}
Jul 27 20:18:43.091891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.091868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p" event={"ID":"1a74d76a-4fff-49b3-96f4-f8605dbace98","Type":"ContainerStarted","Data":"118396c1612f051916de72e8c8bc0bfe1066b8ab454858ae417a6c087030101d"}
Jul 27 20:18:43.094280 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.094204    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-552n5" event={"ID":"f5c1f8bc-723b-4b13-a86c-93c3f0037210","Type":"ContainerStarted","Data":"f73a7add5bb1760aafd757d20271a33e110a3d54beed3fbb89138f5b0b0c4107"}
Jul 27 20:18:43.094280 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.094247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-552n5" event={"ID":"f5c1f8bc-723b-4b13-a86c-93c3f0037210","Type":"ContainerStarted","Data":"ebf22a254f680ef358ac03824c6a82fe1dfa9eab132298d3de516af9caa800b4"}
Jul 27 20:18:43.094490 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.094471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:43.107583 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.107541    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xnms9" podStartSLOduration=1.270996077 podStartE2EDuration="7.10752205s" totalImagesPullingTime="5.836525973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:43.106517167 +0000 UTC m=+55.956283996" watchObservedRunningTime="2026-07-27 20:18:43.10752205 +0000 UTC m=+55.957288874"
Jul 27 20:18:43.123031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:43.122707    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-552n5" podStartSLOduration=1.587965705 podStartE2EDuration="7.122691837s" totalImagesPullingTime="5.534726132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:43.122257558 +0000 UTC m=+55.972024389" watchObservedRunningTime="2026-07-27 20:18:43.122691837 +0000 UTC m=+55.972458664"
Jul 27 20:18:44.703360 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.703326    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-rgnjp"]
Jul 27 20:18:44.709900 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.709872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:18:44.713052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.712877    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6fjcl\""
Jul 27 20:18:44.713052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.712877    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:18:44.713244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.712877    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:18:44.715905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.715874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rgnjp"]
Jul 27 20:18:44.776530 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.776485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzs6\" (UniqueName: \"kubernetes.io/projected/0bf431f0-58e7-40a6-8714-5a666b9a2f5a-kube-api-access-8dzs6\") pod \"downloads-7c97dc4555-rgnjp\" (UID: \"0bf431f0-58e7-40a6-8714-5a666b9a2f5a\") " pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:18:44.877725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.877682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzs6\" (UniqueName: \"kubernetes.io/projected/0bf431f0-58e7-40a6-8714-5a666b9a2f5a-kube-api-access-8dzs6\") pod \"downloads-7c97dc4555-rgnjp\" (UID: \"0bf431f0-58e7-40a6-8714-5a666b9a2f5a\") " pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:18:44.886509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:44.886480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzs6\" (UniqueName: \"kubernetes.io/projected/0bf431f0-58e7-40a6-8714-5a666b9a2f5a-kube-api-access-8dzs6\") pod \"downloads-7c97dc4555-rgnjp\" (UID: \"0bf431f0-58e7-40a6-8714-5a666b9a2f5a\") " pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:18:45.020442 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.020357    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:18:45.102522 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.102319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p" event={"ID":"1a74d76a-4fff-49b3-96f4-f8605dbace98","Type":"ContainerStarted","Data":"67fcb9fda67dadb42a9a8c637df6db8474f1253df94cd19607571282e54d119a"}
Jul 27 20:18:45.104499 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.103232    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:45.109415 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.109388    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p"
Jul 27 20:18:45.119749 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.119694    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-frw9p" podStartSLOduration=4.031072061 podStartE2EDuration="5.119678776s" totalImagesPullingTime="1.088606715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:45.118850199 +0000 UTC m=+57.968617041" watchObservedRunningTime="2026-07-27 20:18:45.119678776 +0000 UTC m=+57.969445585"
Jul 27 20:18:45.384523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.384479    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"]
Jul 27 20:18:45.392732 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.392698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.395545 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.395517    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2psr4\""
Jul 27 20:18:45.395710 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.395574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 20:18:45.396889 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.396784    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:18:45.397748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.397054    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:18:45.397748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.397138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 20:18:45.397748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.397234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:18:45.398616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.398594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"]
Jul 27 20:18:45.481863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.481804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.482021 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.481880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.482021 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.481946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35733880-6d3d-4469-b635-d23a80352b65-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.482157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.482042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpzb\" (UniqueName: \"kubernetes.io/projected/35733880-6d3d-4469-b635-d23a80352b65-kube-api-access-tkpzb\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.582618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.582583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.582618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.582625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.582918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.582661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35733880-6d3d-4469-b635-d23a80352b65-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.582918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.582711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpzb\" (UniqueName: \"kubernetes.io/projected/35733880-6d3d-4469-b635-d23a80352b65-kube-api-access-tkpzb\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.583418 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.583390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35733880-6d3d-4469-b635-d23a80352b65-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.585011 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.584985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.585096 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.585021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35733880-6d3d-4469-b635-d23a80352b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.591291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.591266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpzb\" (UniqueName: \"kubernetes.io/projected/35733880-6d3d-4469-b635-d23a80352b65-kube-api-access-tkpzb\") pod \"prometheus-operator-84d69f794d-zgfqw\" (UID: \"35733880-6d3d-4469-b635-d23a80352b65\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:45.704890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:45.704791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"
Jul 27 20:18:46.401664 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:46.401547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zgfqw"]
Jul 27 20:18:46.404366 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:46.404330    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35733880_6d3d_4469_b635_d23a80352b65.slice/crio-2dc7a7a49a39c72676d17759059151dd5e6fe1075fbbfde5b81af937bdce2b27 WatchSource:0}: Error finding container 2dc7a7a49a39c72676d17759059151dd5e6fe1075fbbfde5b81af937bdce2b27: Status 404 returned error can't find the container with id 2dc7a7a49a39c72676d17759059151dd5e6fe1075fbbfde5b81af937bdce2b27
Jul 27 20:18:46.414254 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:46.414227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rgnjp"]
Jul 27 20:18:46.417267 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:46.417191    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf431f0_58e7_40a6_8714_5a666b9a2f5a.slice/crio-2c841b6a1c1286beba3bb3f0f37a8f578693480fbf22155d758b8acde5821603 WatchSource:0}: Error finding container 2c841b6a1c1286beba3bb3f0f37a8f578693480fbf22155d758b8acde5821603: Status 404 returned error can't find the container with id 2c841b6a1c1286beba3bb3f0f37a8f578693480fbf22155d758b8acde5821603
Jul 27 20:18:47.114139 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:47.114101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw" event={"ID":"35733880-6d3d-4469-b635-d23a80352b65","Type":"ContainerStarted","Data":"2dc7a7a49a39c72676d17759059151dd5e6fe1075fbbfde5b81af937bdce2b27"}
Jul 27 20:18:47.120630 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:47.120595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwgnd" event={"ID":"012af03a-194e-44ab-9e01-54e6427580d2","Type":"ContainerStarted","Data":"2041e4ebc0864c803a8cc48949b4a4ed4463586a58311e8cafe0336062753a79"}
Jul 27 20:18:47.123038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:47.123002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rgnjp" event={"ID":"0bf431f0-58e7-40a6-8714-5a666b9a2f5a","Type":"ContainerStarted","Data":"2c841b6a1c1286beba3bb3f0f37a8f578693480fbf22155d758b8acde5821603"}
Jul 27 20:18:48.471322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:48.471298    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nk775"
Jul 27 20:18:49.132574 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:49.132536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw" event={"ID":"35733880-6d3d-4469-b635-d23a80352b65","Type":"ContainerStarted","Data":"e89b659c1816812c87d0ecf1e8302dc15c4bd0690a00ee53f82930916bfe9580"}
Jul 27 20:18:51.141964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:51.141924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw" event={"ID":"35733880-6d3d-4469-b635-d23a80352b65","Type":"ContainerStarted","Data":"d1a0190249b97925c0186d04eac807c44d49d6fd8b2bb1786b17a865f7e9b059"}
Jul 27 20:18:51.144372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:51.144324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fwgnd" event={"ID":"012af03a-194e-44ab-9e01-54e6427580d2","Type":"ContainerStarted","Data":"f97bf7969883fd24d346a94dffa1998c6e5e579db39fb04963e62ac10a230f9f"}
Jul 27 20:18:51.159828 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:51.159757    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-zgfqw" podStartSLOduration=4.241661514 podStartE2EDuration="6.159737672s" totalImagesPullingTime="1.918076158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:51.159333504 +0000 UTC m=+64.009100347" watchObservedRunningTime="2026-07-27 20:18:51.159737672 +0000 UTC m=+64.009504502"
Jul 27 20:18:51.180033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:51.179976    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fwgnd" podStartSLOduration=1.9752113709999999 podStartE2EDuration="15.17995536s" totalImagesPullingTime="13.204743989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:18:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:51.178752826 +0000 UTC m=+64.028519667" watchObservedRunningTime="2026-07-27 20:18:51.17995536 +0000 UTC m=+64.029722192"
Jul 27 20:18:52.341926 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.341890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:52.341926 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.341930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:52.344931 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.344905    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:18:52.345076 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.345057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:18:52.354650 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.354625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c95815a-2734-4e41-82bf-bd4d0f5883f2-original-pull-secret\") pod \"global-pull-secret-syncer-npmtt\" (UID: \"3c95815a-2734-4e41-82bf-bd4d0f5883f2\") " pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:52.354650 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.354635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da0167a-bfd5-4791-a03c-52fc3795d410-metrics-certs\") pod \"network-metrics-daemon-gvsbh\" (UID: \"1da0167a-bfd5-4791-a03c-52fc3795d410\") " pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:52.443351 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.443303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:52.446150 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.446122    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:18:52.448210 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.448182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npmtt"
Jul 27 20:18:52.457259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.457055    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:18:52.466827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.466784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvk7\""
Jul 27 20:18:52.466992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.466965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26vg\" (UniqueName: \"kubernetes.io/projected/f727887c-fe02-4346-8277-7a51f96728f6-kube-api-access-w26vg\") pod \"network-check-target-t7llv\" (UID: \"f727887c-fe02-4346-8277-7a51f96728f6\") " pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:52.474504 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.474481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvsbh"
Jul 27 20:18:52.596658 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.596624    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npmtt"]
Jul 27 20:18:52.599664 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:52.599633    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c95815a_2734_4e41_82bf_bd4d0f5883f2.slice/crio-5856a774f87c1808fc3186881461c4c4bd07cea2afb1f68a3fe811c5813a3f37 WatchSource:0}: Error finding container 5856a774f87c1808fc3186881461c4c4bd07cea2afb1f68a3fe811c5813a3f37: Status 404 returned error can't find the container with id 5856a774f87c1808fc3186881461c4c4bd07cea2afb1f68a3fe811c5813a3f37
Jul 27 20:18:52.624400 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.624370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvsbh"]
Jul 27 20:18:52.628882 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:52.628852    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da0167a_bfd5_4791_a03c_52fc3795d410.slice/crio-44d615aef8c7f3982d1114f24f88df14146fb45cf3696f4cc58808aec1d2362e WatchSource:0}: Error finding container 44d615aef8c7f3982d1114f24f88df14146fb45cf3696f4cc58808aec1d2362e: Status 404 returned error can't find the container with id 44d615aef8c7f3982d1114f24f88df14146fb45cf3696f4cc58808aec1d2362e
Jul 27 20:18:52.738861 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.738830    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"]
Jul 27 20:18:52.741610 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.741586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.748079 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.748049    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xfgdx"]
Jul 27 20:18:52.749367 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.749343    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 20:18:52.749840 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.749646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:18:52.749840 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.749747    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rxh8x\""
Jul 27 20:18:52.750311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.750062    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.752555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.752510    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:18:52.753049 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.752885    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:18:52.753049 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.752954    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:18:52.753049 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.752885    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bjbdd\""
Jul 27 20:18:52.755157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.755082    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"]
Jul 27 20:18:52.759199 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.759181    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xjfvv\""
Jul 27 20:18:52.767553 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.767535    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:18:52.776493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.776470    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"]
Jul 27 20:18:52.779120 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.779098    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.782036 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.782011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:18:52.782137 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.782104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 20:18:52.782201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.782105    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xw69d\""
Jul 27 20:18:52.782201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.782140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 20:18:52.791101 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.791067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"]
Jul 27 20:18:52.848195 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848119    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848195 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-wtmp\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-sys\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-metrics-client-ca\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bzf\" (UniqueName: \"kubernetes.io/projected/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-api-access-p9bzf\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.848377 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-root\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-textfile\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0338909-e8b6-4b62-aa8a-e581a44479dd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848476    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848503    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjhk\" (UniqueName: \"kubernetes.io/projected/a1f77da5-3a70-4181-87e5-91b79fa039b4-kube-api-access-grjhk\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4kk\" (UniqueName: \"kubernetes.io/projected/6c03c33a-592b-41ab-8f41-f5aad4f3791a-kube-api-access-dx4kk\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.848689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c33a-592b-41ab-8f41-f5aad4f3791a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.849022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.848739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.911993 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.911957    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7llv"]
Jul 27 20:18:52.914725 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:52.914698    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf727887c_fe02_4346_8277_7a51f96728f6.slice/crio-264f8cf6242ed838c5e19fea4ad47d2d4c43df51fcf4f42a6826c3c2fa5d3b26 WatchSource:0}: Error finding container 264f8cf6242ed838c5e19fea4ad47d2d4c43df51fcf4f42a6826c3c2fa5d3b26: Status 404 returned error can't find the container with id 264f8cf6242ed838c5e19fea4ad47d2d4c43df51fcf4f42a6826c3c2fa5d3b26
Jul 27 20:18:52.950016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.949983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-wtmp\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-sys\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-metrics-client-ca\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-sys\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bzf\" (UniqueName: \"kubernetes.io/projected/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-api-access-p9bzf\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950201 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950437 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-wtmp\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950437 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-root\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950437 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1f77da5-3a70-4181-87e5-91b79fa039b4-root\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950437 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.950347    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 27 20:18:52.950437 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.950408    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls podName:c0338909-e8b6-4b62-aa8a-e581a44479dd nodeName:}" failed. No retries permitted until 2026-07-27 20:18:53.450386345 +0000 UTC m=+66.300153157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-qkb8d" (UID: "c0338909-e8b6-4b62-aa8a-e581a44479dd") : secret "kube-state-metrics-tls" not found
Jul 27 20:18:52.950669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-textfile\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0338909-e8b6-4b62-aa8a-e581a44479dd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjhk\" (UniqueName: \"kubernetes.io/projected/a1f77da5-3a70-4181-87e5-91b79fa039b4-kube-api-access-grjhk\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4kk\" (UniqueName: \"kubernetes.io/projected/6c03c33a-592b-41ab-8f41-f5aad4f3791a-kube-api-access-dx4kk\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c33a-592b-41ab-8f41-f5aad4f3791a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-textfile\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-metrics-client-ca\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.950928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.951285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.951285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.950964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.951285 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.951109    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 20:18:52.951285 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.951155    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls podName:a1f77da5-3a70-4181-87e5-91b79fa039b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:53.45113825 +0000 UTC m=+66.300905072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls") pod "node-exporter-xfgdx" (UID: "a1f77da5-3a70-4181-87e5-91b79fa039b4") : secret "node-exporter-tls" not found
Jul 27 20:18:52.951477 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.951342    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 20:18:52.951477 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.951381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.951477 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:52.951391    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls podName:6c03c33a-592b-41ab-8f41-f5aad4f3791a nodeName:}" failed. No retries permitted until 2026-07-27 20:18:53.45137416 +0000 UTC m=+66.301140984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-pfnqb" (UID: "6c03c33a-592b-41ab-8f41-f5aad4f3791a") : secret "openshift-state-metrics-tls" not found
Jul 27 20:18:52.951713 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.951692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0338909-e8b6-4b62-aa8a-e581a44479dd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.951910 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.951857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.952121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.952098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.952207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.952185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c03c33a-592b-41ab-8f41-f5aad4f3791a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.954180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.954152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.954522 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.954497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:52.954773 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.954750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.962919 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.962892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjhk\" (UniqueName: \"kubernetes.io/projected/a1f77da5-3a70-4181-87e5-91b79fa039b4-kube-api-access-grjhk\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:52.963104 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.963017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bzf\" (UniqueName: \"kubernetes.io/projected/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-api-access-p9bzf\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:52.965012 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:52.964989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4kk\" (UniqueName: \"kubernetes.io/projected/6c03c33a-592b-41ab-8f41-f5aad4f3791a-kube-api-access-dx4kk\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:53.100528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.100379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-552n5"
Jul 27 20:18:53.154824 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.154777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvsbh" event={"ID":"1da0167a-bfd5-4791-a03c-52fc3795d410","Type":"ContainerStarted","Data":"44d615aef8c7f3982d1114f24f88df14146fb45cf3696f4cc58808aec1d2362e"}
Jul 27 20:18:53.156543 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.156509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npmtt" event={"ID":"3c95815a-2734-4e41-82bf-bd4d0f5883f2","Type":"ContainerStarted","Data":"5856a774f87c1808fc3186881461c4c4bd07cea2afb1f68a3fe811c5813a3f37"}
Jul 27 20:18:53.158131 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.158068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7llv" event={"ID":"f727887c-fe02-4346-8277-7a51f96728f6","Type":"ContainerStarted","Data":"264f8cf6242ed838c5e19fea4ad47d2d4c43df51fcf4f42a6826c3c2fa5d3b26"}
Jul 27 20:18:53.457169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.456545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:53.457169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.456622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:53.457169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.456680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:53.466026 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.465966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c03c33a-592b-41ab-8f41-f5aad4f3791a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pfnqb\" (UID: \"6c03c33a-592b-41ab-8f41-f5aad4f3791a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:53.466572 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.466518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1f77da5-3a70-4181-87e5-91b79fa039b4-node-exporter-tls\") pod \"node-exporter-xfgdx\" (UID: \"a1f77da5-3a70-4181-87e5-91b79fa039b4\") " pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:53.466956 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.466912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0338909-e8b6-4b62-aa8a-e581a44479dd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-qkb8d\" (UID: \"c0338909-e8b6-4b62-aa8a-e581a44479dd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:53.653450 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.653384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"
Jul 27 20:18:53.661725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.661700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xfgdx"
Jul 27 20:18:53.690855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.690791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"
Jul 27 20:18:53.748267 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:53.748226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f77da5_3a70_4181_87e5_91b79fa039b4.slice/crio-43e614e64a3fa6420d916dfddad35d2f54cb5f25a6276ce17832ee2eca728e39 WatchSource:0}: Error finding container 43e614e64a3fa6420d916dfddad35d2f54cb5f25a6276ce17832ee2eca728e39: Status 404 returned error can't find the container with id 43e614e64a3fa6420d916dfddad35d2f54cb5f25a6276ce17832ee2eca728e39
Jul 27 20:18:53.826512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.826108    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:18:53.835246 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.833972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.837454    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.837719    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.838032    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.838235    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.838532    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.838800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.839044    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.839224    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.839404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gq4qb\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.839582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 20:18:53.843239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.843042    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:18:53.849884 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.848621    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:18:53.853248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.853230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.858964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.857868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:18:53.858964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.858140    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:18:53.858964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.858334    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vmj8n\""
Jul 27 20:18:53.858964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.858545    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:18:53.858964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.858765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.861292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.860657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.861958    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:18:53.864715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:18:53.864715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6tf\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.864934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.864906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.889328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.889290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d"]
Jul 27 20:18:53.914260 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:53.913390    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0338909_e8b6_4b62_aa8a_e581a44479dd.slice/crio-e625d7a8fbfa93619a62e888283c4b4f068ab4c9a52dc3a72738fa6f3fff2302 WatchSource:0}: Error finding container e625d7a8fbfa93619a62e888283c4b4f068ab4c9a52dc3a72738fa6f3fff2302: Status 404 returned error can't find the container with id e625d7a8fbfa93619a62e888283c4b4f068ab4c9a52dc3a72738fa6f3fff2302
Jul 27 20:18:53.922488 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.922395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb"]
Jul 27 20:18:53.929047 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:53.929016    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c03c33a_592b_41ab_8f41_f5aad4f3791a.slice/crio-73f4d298f0b0a270636c3ae0c609d62f56f5dd95d19a7990f4dce21f4638855f WatchSource:0}: Error finding container 73f4d298f0b0a270636c3ae0c609d62f56f5dd95d19a7990f4dce21f4638855f: Status 404 returned error can't find the container with id 73f4d298f0b0a270636c3ae0c609d62f56f5dd95d19a7990f4dce21f4638855f
Jul 27 20:18:53.965639 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.965757 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.965757 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwbf\" (UniqueName: \"kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.965757 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.965757 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.966206 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.966206 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.966206 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.966206 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968036 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.965908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6tf\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968036 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.967011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.968794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.969217 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.968825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:53.969217 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:53.968946    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 27 20:18:53.969217 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:53.969014    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls podName:3a84902a-4439-4e01-9552-475e1127a225 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:54.468984193 +0000 UTC m=+67.318751005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3a84902a-4439-4e01-9552-475e1127a225") : secret "alertmanager-main-tls" not found
Jul 27 20:18:53.970159 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:18:53.970103    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle podName:3a84902a-4439-4e01-9552-475e1127a225 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:54.470084979 +0000 UTC m=+67.319851811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3a84902a-4439-4e01-9552-475e1127a225") : configmap references non-existent config key: ca-bundle.crt
Jul 27 20:18:53.970652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.970602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.973498 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.973476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.973717 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.973695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.975275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.975190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.975896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.975877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.978541 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.978485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.978541 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.978490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.978981 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.978939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6tf\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.979173 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.979152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:53.983937 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:53.983896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:54.069279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.069419 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwbf\" (UniqueName: \"kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.069419 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.069419 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.069579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.069579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.069469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.070283 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.070256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.070847 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.070775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.070983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.070962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.072955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.072901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.073302 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.073258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.081521 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.081501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwbf\" (UniqueName: \"kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf\") pod \"console-5b5d446c6b-9qm4h\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") " pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.169148 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.168959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb" event={"ID":"6c03c33a-592b-41ab-8f41-f5aad4f3791a","Type":"ContainerStarted","Data":"5a9a9d8678137cda892fa164d45308b414187059ef59ea3e06384f27462c6245"}
Jul 27 20:18:54.169148 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.169068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb" event={"ID":"6c03c33a-592b-41ab-8f41-f5aad4f3791a","Type":"ContainerStarted","Data":"12e1424701f0b9247184f4d226bcfba75f5f26534fe4fe99ff2379dc52b3c5e7"}
Jul 27 20:18:54.169148 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.169099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb" event={"ID":"6c03c33a-592b-41ab-8f41-f5aad4f3791a","Type":"ContainerStarted","Data":"73f4d298f0b0a270636c3ae0c609d62f56f5dd95d19a7990f4dce21f4638855f"}
Jul 27 20:18:54.175314 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.174867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvsbh" event={"ID":"1da0167a-bfd5-4791-a03c-52fc3795d410","Type":"ContainerStarted","Data":"0ee38442d3bc6278401d8eaed757c425847e085b0ea39400705997c2953ce378"}
Jul 27 20:18:54.175314 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.174928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvsbh" event={"ID":"1da0167a-bfd5-4791-a03c-52fc3795d410","Type":"ContainerStarted","Data":"bd0c9194d23533d850e80d074a93a1a4d672b5910b74dcbee1d75e70ea909e86"}
Jul 27 20:18:54.177569 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.177540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfgdx" event={"ID":"a1f77da5-3a70-4181-87e5-91b79fa039b4","Type":"ContainerStarted","Data":"43e614e64a3fa6420d916dfddad35d2f54cb5f25a6276ce17832ee2eca728e39"}
Jul 27 20:18:54.180155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.180087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d" event={"ID":"c0338909-e8b6-4b62-aa8a-e581a44479dd","Type":"ContainerStarted","Data":"e625d7a8fbfa93619a62e888283c4b4f068ab4c9a52dc3a72738fa6f3fff2302"}
Jul 27 20:18:54.232077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.231789    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:18:54.410607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.410025    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gvsbh" podStartSLOduration=66.288302714 podStartE2EDuration="1m7.409999633s" totalImagesPullingTime="1.121696919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:54.193989356 +0000 UTC m=+67.043756188" watchObservedRunningTime="2026-07-27 20:18:54.409999633 +0000 UTC m=+67.259766464"
Jul 27 20:18:54.411568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.411502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:18:54.415524 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:54.415494    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9715601_a2cf_44eb_8eb4_862ccffde093.slice/crio-4d027bddb26599a6ababfe7c75ead9e8b0e679f01702f0819e322a5c932b0991 WatchSource:0}: Error finding container 4d027bddb26599a6ababfe7c75ead9e8b0e679f01702f0819e322a5c932b0991: Status 404 returned error can't find the container with id 4d027bddb26599a6ababfe7c75ead9e8b0e679f01702f0819e322a5c932b0991
Jul 27 20:18:54.473858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.473637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:54.473858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.473738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:54.474783 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.474728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:54.478050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.478006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:54.756083 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:54.756043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:18:55.193801 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.193737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5d446c6b-9qm4h" event={"ID":"b9715601-a2cf-44eb-8eb4-862ccffde093","Type":"ContainerStarted","Data":"4d027bddb26599a6ababfe7c75ead9e8b0e679f01702f0819e322a5c932b0991"}
Jul 27 20:18:55.290479 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.290376    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:18:55.435672 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:18:55.435627    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a84902a_4439_4e01_9552_475e1127a225.slice/crio-8892995dc06bda0a259daece7dd37b99fafb27a365b190bf0a76785e27945a78 WatchSource:0}: Error finding container 8892995dc06bda0a259daece7dd37b99fafb27a365b190bf0a76785e27945a78: Status 404 returned error can't find the container with id 8892995dc06bda0a259daece7dd37b99fafb27a365b190bf0a76785e27945a78
Jul 27 20:18:55.712284 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.712198    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84544c7c78-s2pqk"]
Jul 27 20:18:55.728771 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.727999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.732218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r27l7\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.732452    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.732472    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-70u5s0mjtpt2t\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.732631    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.732831    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.733040    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 20:18:55.734638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.733256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 20:18:55.740344 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.739908    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84544c7c78-s2pqk"]
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.787850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ebdc5-e571-41f7-ab58-9afd88a329e6-metrics-client-ca\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.787911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.787940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-grpc-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.787967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.788004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftkk\" (UniqueName: \"kubernetes.io/projected/da6ebdc5-e571-41f7-ab58-9afd88a329e6-kube-api-access-rftkk\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.788031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.788060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.788279 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.788105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.888927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.888886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftkk\" (UniqueName: \"kubernetes.io/projected/da6ebdc5-e571-41f7-ab58-9afd88a329e6-kube-api-access-rftkk\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.888938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.888970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ebdc5-e571-41f7-ab58-9afd88a329e6-metrics-client-ca\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889144    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-grpc-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.889373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.890710 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.889972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ebdc5-e571-41f7-ab58-9afd88a329e6-metrics-client-ca\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.893659 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.893124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.896946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.894406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-grpc-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.896946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.895014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.896946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.895038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-tls\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.896946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.895574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.897258 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.897199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da6ebdc5-e571-41f7-ab58-9afd88a329e6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:55.897340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:55.897298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftkk\" (UniqueName: \"kubernetes.io/projected/da6ebdc5-e571-41f7-ab58-9afd88a329e6-kube-api-access-rftkk\") pod \"thanos-querier-84544c7c78-s2pqk\" (UID: \"da6ebdc5-e571-41f7-ab58-9afd88a329e6\") " pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:56.044254 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:56.043898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:18:56.200762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:56.200661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"8892995dc06bda0a259daece7dd37b99fafb27a365b190bf0a76785e27945a78"}
Jul 27 20:18:57.039459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.039410    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b795b9688-fmvnd"]
Jul 27 20:18:57.042207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.042176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.045138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.045075    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 20:18:57.045281 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.045114    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:18:57.046157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.046089    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 20:18:57.046470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.046443    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t64hr\""
Jul 27 20:18:57.046579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.046491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 20:18:57.046579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.046445    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fkgb03qob2hd5\""
Jul 27 20:18:57.058248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.058214    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b795b9688-fmvnd"]
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-tls\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/90244c4a-8765-4fab-8c5a-b03368dfb9fa-audit-log\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-client-certs\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-client-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.100740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4w54\" (UniqueName: \"kubernetes.io/projected/90244c4a-8765-4fab-8c5a-b03368dfb9fa-kube-api-access-n4w54\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.101208 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.100862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-metrics-server-audit-profiles\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-client-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4w54\" (UniqueName: \"kubernetes.io/projected/90244c4a-8765-4fab-8c5a-b03368dfb9fa-kube-api-access-n4w54\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-metrics-server-audit-profiles\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-tls\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.201974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/90244c4a-8765-4fab-8c5a-b03368dfb9fa-audit-log\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.202009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-client-certs\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202658 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.202496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/90244c4a-8765-4fab-8c5a-b03368dfb9fa-audit-log\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.202658 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.202644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.203114 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.203020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/90244c4a-8765-4fab-8c5a-b03368dfb9fa-metrics-server-audit-profiles\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.205747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.205719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-tls\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.206373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.206286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-secret-metrics-server-client-certs\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.207267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.207224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/90244c4a-8765-4fab-8c5a-b03368dfb9fa-client-ca-bundle\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.211623 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.211583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4w54\" (UniqueName: \"kubernetes.io/projected/90244c4a-8765-4fab-8c5a-b03368dfb9fa-kube-api-access-n4w54\") pod \"metrics-server-5b795b9688-fmvnd\" (UID: \"90244c4a-8765-4fab-8c5a-b03368dfb9fa\") " pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.365886 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.365851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:18:57.514214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.514182    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"]
Jul 27 20:18:57.517008 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.516983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.520072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.519854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 20:18:57.520072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.520053    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-44ztz\""
Jul 27 20:18:57.525175 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.525151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"]
Jul 27 20:18:57.606199 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.606171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6d08629-b41d-47e4-b1ac-110fcd2862ba-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.606366 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.606220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnxf\" (UniqueName: \"kubernetes.io/projected/a6d08629-b41d-47e4-b1ac-110fcd2862ba-kube-api-access-brnxf\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.707903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.707761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6d08629-b41d-47e4-b1ac-110fcd2862ba-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.707903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.707876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnxf\" (UniqueName: \"kubernetes.io/projected/a6d08629-b41d-47e4-b1ac-110fcd2862ba-kube-api-access-brnxf\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.711002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.710971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a6d08629-b41d-47e4-b1ac-110fcd2862ba-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.719151 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.719116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnxf\" (UniqueName: \"kubernetes.io/projected/a6d08629-b41d-47e4-b1ac-110fcd2862ba-kube-api-access-brnxf\") pod \"monitoring-plugin-68b7fc9574-dhtz4\" (UID: \"a6d08629-b41d-47e4-b1ac-110fcd2862ba\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:57.833057 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:57.833028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:18:58.938835 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.938781    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:18:58.942312 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.942160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:58.945277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945249    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:18:58.945429 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:18:58.945673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945654    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:18:58.945800 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:18:58.945872 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945837    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:18:58.945929 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dqc72skpp8c9n\""
Jul 27 20:18:58.945974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.945787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:18:58.948603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.946514    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jvlxj\""
Jul 27 20:18:58.948603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.946928    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:18:58.948603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.947187    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:18:58.948603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.947401    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:18:58.948603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.948539    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:18:58.949009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.948716    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:18:58.949009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.948728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:18:58.949009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.948953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:18:58.950979 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.950954    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:18:58.958679 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:58.958617    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022539    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.022784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.022968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kgf\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.023340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.023276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kgf\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.126318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.124999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.125027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.125048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.126254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.126514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.126741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.126894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.127320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.126970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.128126 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.128098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.128316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.128287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.131944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.131879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.133577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.133518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.133692 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.133594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.134297 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.134254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.134900 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.134860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.135476 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.135440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.135615 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.135581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.136338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.134252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.137001 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.136979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.137001 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.136991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.137308 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.137288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.138715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.138683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kgf\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.139341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.139295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:18:59.260374 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:18:59.260281    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:19:02.338420 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:02.338337    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:19:06.639104 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:06.639072    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84544c7c78-s2pqk"]
Jul 27 20:19:08.276147 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:19:08.276110    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ebdc5_e571_41f7_ab58_9afd88a329e6.slice/crio-02071b81b55a6641c38977de159239a58e09082aef96f1276440e0b136dac4d4 WatchSource:0}: Error finding container 02071b81b55a6641c38977de159239a58e09082aef96f1276440e0b136dac4d4: Status 404 returned error can't find the container with id 02071b81b55a6641c38977de159239a58e09082aef96f1276440e0b136dac4d4
Jul 27 20:19:08.724671 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:08.724005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"]
Jul 27 20:19:08.759652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:08.759621    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b795b9688-fmvnd"]
Jul 27 20:19:08.764391 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:19:08.764361    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90244c4a_8765_4fab_8c5a_b03368dfb9fa.slice/crio-5de843c26e23a7027571413e2a37d56e892843d876c6fd14234472f4b0de7428 WatchSource:0}: Error finding container 5de843c26e23a7027571413e2a37d56e892843d876c6fd14234472f4b0de7428: Status 404 returned error can't find the container with id 5de843c26e23a7027571413e2a37d56e892843d876c6fd14234472f4b0de7428
Jul 27 20:19:08.795865 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:08.794104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:19:09.256270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.256232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npmtt" event={"ID":"3c95815a-2734-4e41-82bf-bd4d0f5883f2","Type":"ContainerStarted","Data":"cd5c24627b9f66fd0411abccac2464fb49897d92a0345a71151da803fc1613b5"}
Jul 27 20:19:09.261409 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.261375    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="99356f9144a8f1ccc5dca32cf989284ce6428553f332d05b6d83443686a16106" exitCode=0
Jul 27 20:19:09.261586 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.261456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"99356f9144a8f1ccc5dca32cf989284ce6428553f332d05b6d83443686a16106"}
Jul 27 20:19:09.261586 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.261535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"bc43bfcbbc3b70c63ae6535aa7a84224e83dfef56a493282bf6b334d1f643f0c"}
Jul 27 20:19:09.263655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.263612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd" event={"ID":"90244c4a-8765-4fab-8c5a-b03368dfb9fa","Type":"ContainerStarted","Data":"5de843c26e23a7027571413e2a37d56e892843d876c6fd14234472f4b0de7428"}
Jul 27 20:19:09.270704 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.270577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d" event={"ID":"c0338909-e8b6-4b62-aa8a-e581a44479dd","Type":"ContainerStarted","Data":"a30be6cf6cceae19cc767fc339efb125a3e1eb4f465558aeb080d94c4309d9e3"}
Jul 27 20:19:09.270704 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.270640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d" event={"ID":"c0338909-e8b6-4b62-aa8a-e581a44479dd","Type":"ContainerStarted","Data":"cbc91690673a33228332b35f4896a7aef44cfc74da9b33ec5d0b4014a1f5eba8"}
Jul 27 20:19:09.270704 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.270669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d" event={"ID":"c0338909-e8b6-4b62-aa8a-e581a44479dd","Type":"ContainerStarted","Data":"76fd8c5057d713cd26c911a0bf28ec99f74ad5b79f7381b0796c717376847bf1"}
Jul 27 20:19:09.277307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.275393    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e" exitCode=0
Jul 27 20:19:09.277307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.275495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e"}
Jul 27 20:19:09.285322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.284498    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1f77da5-3a70-4181-87e5-91b79fa039b4" containerID="b76510fa7791c1e59ce016e5c9052b38e583312de4c08ef342acaa0d3c64977c" exitCode=0
Jul 27 20:19:09.285322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.284612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfgdx" event={"ID":"a1f77da5-3a70-4181-87e5-91b79fa039b4","Type":"ContainerDied","Data":"b76510fa7791c1e59ce016e5c9052b38e583312de4c08ef342acaa0d3c64977c"}
Jul 27 20:19:09.293963 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.289910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5d446c6b-9qm4h" event={"ID":"b9715601-a2cf-44eb-8eb4-862ccffde093","Type":"ContainerStarted","Data":"3f3a0ebe14998aa4100e99e4ab538af84dd625f642d6f424d71aeac2420cb04b"}
Jul 27 20:19:09.293963 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.293194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rgnjp" event={"ID":"0bf431f0-58e7-40a6-8714-5a666b9a2f5a","Type":"ContainerStarted","Data":"1014bf166a0f82f9c94baa4cb537adf1c0aac0b3da02f6053222a4562b601bff"}
Jul 27 20:19:09.293963 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.293735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:19:09.297513 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.297394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"02071b81b55a6641c38977de159239a58e09082aef96f1276440e0b136dac4d4"}
Jul 27 20:19:09.298013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.297997    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-rgnjp"
Jul 27 20:19:09.300907 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.300860    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-qkb8d" podStartSLOduration=4.725550638 podStartE2EDuration="17.300842469s" totalImagesPullingTime="12.575291831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.300083577 +0000 UTC m=+82.149850418" watchObservedRunningTime="2026-07-27 20:19:09.300842469 +0000 UTC m=+82.150609301"
Jul 27 20:19:09.301678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.301639    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-npmtt" podStartSLOduration=66.346814417 podStartE2EDuration="1m22.301627011s" totalImagesPullingTime="15.954812594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.273035511 +0000 UTC m=+82.122802342" watchObservedRunningTime="2026-07-27 20:19:09.301627011 +0000 UTC m=+82.151393838"
Jul 27 20:19:09.306687 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.306659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb" event={"ID":"6c03c33a-592b-41ab-8f41-f5aad4f3791a","Type":"ContainerStarted","Data":"561b4fa44f34ac045a6ff6041d1804b42c700622992bf65b9bdcb04027f2cfff"}
Jul 27 20:19:09.309132 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.309070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7llv" event={"ID":"f727887c-fe02-4346-8277-7a51f96728f6","Type":"ContainerStarted","Data":"05016a9602ed40fb80dc662352c28d80ead8a37ac93f02b76ef54522edb217b2"}
Jul 27 20:19:09.311051 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.310944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:19:09.312250 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.312223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4" event={"ID":"a6d08629-b41d-47e4-b1ac-110fcd2862ba","Type":"ContainerStarted","Data":"0c10632f6a62dd0c9dc25705e8bba48246b07f3076c5ec400404879feb4a87fc"}
Jul 27 20:19:09.408024 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.406594    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-rgnjp" podStartSLOduration=3.2537576 podStartE2EDuration="25.406572311s" totalImagesPullingTime="22.152814711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.405635294 +0000 UTC m=+82.255402139" watchObservedRunningTime="2026-07-27 20:19:09.406572311 +0000 UTC m=+82.256339142"
Jul 27 20:19:09.408024 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.407076    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pfnqb" podStartSLOduration=5.037986486 podStartE2EDuration="17.407062921s" totalImagesPullingTime="12.369076435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.381865615 +0000 UTC m=+82.231632447" watchObservedRunningTime="2026-07-27 20:19:09.407062921 +0000 UTC m=+82.256829756"
Jul 27 20:19:09.445050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:09.444972    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7llv" podStartSLOduration=66.818090584 podStartE2EDuration="1m22.44495158s" totalImagesPullingTime="15.626860996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.443215027 +0000 UTC m=+82.292981857" watchObservedRunningTime="2026-07-27 20:19:09.44495158 +0000 UTC m=+82.294718411"
Jul 27 20:19:10.333368 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:10.331442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfgdx" event={"ID":"a1f77da5-3a70-4181-87e5-91b79fa039b4","Type":"ContainerStarted","Data":"ee4ae83a8f6879b7a297eccfb569dde231b9ef29c069cb9febf67fd9f3f82cbc"}
Jul 27 20:19:10.333368 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:10.331503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfgdx" event={"ID":"a1f77da5-3a70-4181-87e5-91b79fa039b4","Type":"ContainerStarted","Data":"26ab61f0affa2714b27bd3b76a185f2aa34d12d310a476e967c9dd6ac3297629"}
Jul 27 20:19:10.359488 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:10.359398    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xfgdx" podStartSLOduration=5.553182378 podStartE2EDuration="18.359374135s" totalImagesPullingTime="12.704322442s" totalInitContainerRuntime="101.869315ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:10.358503838 +0000 UTC m=+83.208270669" watchObservedRunningTime="2026-07-27 20:19:10.359374135 +0000 UTC m=+83.209141000"
Jul 27 20:19:10.360758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:10.360478    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b5d446c6b-9qm4h" podStartSLOduration=3.20570549 podStartE2EDuration="17.360463168s" totalImagesPullingTime="14.154757678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:09.464842061 +0000 UTC m=+82.314608893" watchObservedRunningTime="2026-07-27 20:19:10.360463168 +0000 UTC m=+83.210229999"
Jul 27 20:19:14.232979 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:14.232870    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:19:16.369048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.368850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"2cef93c9a22f73abab750343153b06c53f5ca2aec53e10308f635b368b30b3f1"}
Jul 27 20:19:16.369048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.368939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"eae51b8767c330e1d5aa31ae8791b8b0f925b0848e757a2ff2652f28452d378d"}
Jul 27 20:19:16.369048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.368982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"fe2efefa451534e4ffafc4a15f6f37a68736665b4519f5e76a63c1fea5c7ec36"}
Jul 27 20:19:16.374120 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.371254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4" event={"ID":"a6d08629-b41d-47e4-b1ac-110fcd2862ba","Type":"ContainerStarted","Data":"b85ececd1973495d6a513cbfb4b67dece19e17c13b240aa5985250a4ab6c5e4c"}
Jul 27 20:19:16.374383 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.374357    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:19:16.380092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.379909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"f8e96a94d527d2370d57d7a1f46a41dcf91742ec9716961591e8f97167832bfc"}
Jul 27 20:19:16.380092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.379992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"16daa68759c840266c7ecb3cec57cd3317a60db46ff797d2eafcf4fac3774807"}
Jul 27 20:19:16.380092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.380032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"e125b9e323204138cfe0ee4f42df1aef4646b07586ce78daaa8673c04ed9c710"}
Jul 27 20:19:16.382055 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.381993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd" event={"ID":"90244c4a-8765-4fab-8c5a-b03368dfb9fa","Type":"ContainerStarted","Data":"efbb1b52c80ed481745e15685b850548b4d86cf0f86f9f18bbce6a910ab8d6a6"}
Jul 27 20:19:16.386677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.386622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4"
Jul 27 20:19:16.387249 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.387219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"}
Jul 27 20:19:16.387352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.387296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"}
Jul 27 20:19:16.387352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.387337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"}
Jul 27 20:19:16.388913 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.388860    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-dhtz4" podStartSLOduration=12.386057925 podStartE2EDuration="19.388843642s" totalImagesPullingTime="7.002785717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:16.388229243 +0000 UTC m=+89.237996075" watchObservedRunningTime="2026-07-27 20:19:16.388843642 +0000 UTC m=+89.238610473"
Jul 27 20:19:16.430443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:16.424637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd" podStartSLOduration=12.444517321 podStartE2EDuration="19.424611096s" totalImagesPullingTime="6.980093775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:16.423933358 +0000 UTC m=+89.273700190" watchObservedRunningTime="2026-07-27 20:19:16.424611096 +0000 UTC m=+89.274377927"
Jul 27 20:19:17.367001 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.366957    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:19:17.367262 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.367070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:19:17.394898 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.394857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"2931a2706594021cd0af6693177a16592796dc873047b34b32ff408768af7281"}
Jul 27 20:19:17.395295 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.394941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"c350465cc200318e143c45622a5085c057eb53b0d7ec5ed30985d2709e38681f"}
Jul 27 20:19:17.395295 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.394969    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerStarted","Data":"64459fe54f560bd449b47294be73175632807599339efd22433cfd570bdff333"}
Jul 27 20:19:17.397989 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.397959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"}
Jul 27 20:19:17.398132 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.398022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"}
Jul 27 20:19:17.428213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:17.428159    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=12.889602382 podStartE2EDuration="19.428142392s" totalImagesPullingTime="6.49346117s" totalInitContainerRuntime="45.07884ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:17.425136423 +0000 UTC m=+90.274903255" watchObservedRunningTime="2026-07-27 20:19:17.428142392 +0000 UTC m=+90.277909223"
Jul 27 20:19:18.405454 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.405414    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"e154eee00fc05b7c5354a86fb1763d12f9a7b997f3f134fdd4bda5fe8bd0fcaf"}
Jul 27 20:19:18.406058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.405499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"91e59c9798e6cae830e27a8cbb82de141f57d9971da1515221b0b979f24f531e"}
Jul 27 20:19:18.406058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.405557    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:19:18.406058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.405601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" event={"ID":"da6ebdc5-e571-41f7-ab58-9afd88a329e6","Type":"ContainerStarted","Data":"01654231260e876b921480fb3c38e55208f0dd471979a32f8451a692e788928b"}
Jul 27 20:19:18.408652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.408626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerStarted","Data":"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"}
Jul 27 20:19:18.479967 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.479862    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.454844068 podStartE2EDuration="25.479838923s" totalImagesPullingTime="21.013542883s" totalInitContainerRuntime="11.451972ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:18.478488772 +0000 UTC m=+91.328255627" watchObservedRunningTime="2026-07-27 20:19:18.479838923 +0000 UTC m=+91.329605755"
Jul 27 20:19:18.481783 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:18.481737    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk" podStartSLOduration=14.691210559 podStartE2EDuration="23.481718784s" totalImagesPullingTime="8.790508225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:18.44176582 +0000 UTC m=+91.291532700" watchObservedRunningTime="2026-07-27 20:19:18.481718784 +0000 UTC m=+91.331485616"
Jul 27 20:19:19.261138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:19.261092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:19:24.419974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:24.419945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84544c7c78-s2pqk"
Jul 27 20:19:34.322738 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.322673    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b5d446c6b-9qm4h" podUID="b9715601-a2cf-44eb-8eb4-862ccffde093" containerName="console" containerID="cri-o://3f3a0ebe14998aa4100e99e4ab538af84dd625f642d6f424d71aeac2420cb04b" gracePeriod=15
Jul 27 20:19:34.476405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.476371    2560 generic.go:356] "Generic (PLEG): container finished" podID="b9715601-a2cf-44eb-8eb4-862ccffde093" containerID="3f3a0ebe14998aa4100e99e4ab538af84dd625f642d6f424d71aeac2420cb04b" exitCode=0
Jul 27 20:19:34.476598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.476457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5d446c6b-9qm4h" event={"ID":"b9715601-a2cf-44eb-8eb4-862ccffde093","Type":"ContainerDied","Data":"3f3a0ebe14998aa4100e99e4ab538af84dd625f642d6f424d71aeac2420cb04b"}
Jul 27 20:19:34.587671 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.587599    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:19:34.673214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673179    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673585 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673403    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxwbf\" (UniqueName: \"kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673585 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673453    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca\") pod \"b9715601-a2cf-44eb-8eb4-862ccffde093\" (UID: \"b9715601-a2cf-44eb-8eb4-862ccffde093\") "
Jul 27 20:19:34.673718 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config" (OuterVolumeSpecName: "console-config") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:19:34.673772 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673735    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:19:34.673864 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673836    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca" (OuterVolumeSpecName: "service-ca") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:19:34.673916 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673899    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-console-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:34.673951 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.673920    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-oauth-serving-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:34.675764 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.675728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:19:34.675903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.675777    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf" (OuterVolumeSpecName: "kube-api-access-fxwbf") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "kube-api-access-fxwbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:19:34.675903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.675826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b9715601-a2cf-44eb-8eb4-862ccffde093" (UID: "b9715601-a2cf-44eb-8eb4-862ccffde093"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:19:34.775205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.775166    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxwbf\" (UniqueName: \"kubernetes.io/projected/b9715601-a2cf-44eb-8eb4-862ccffde093-kube-api-access-fxwbf\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:34.775205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.775197    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9715601-a2cf-44eb-8eb4-862ccffde093-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:34.775205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.775208    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-oauth-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:34.775205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:34.775218    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9715601-a2cf-44eb-8eb4-862ccffde093-console-serving-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:19:35.483974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.483944    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5d446c6b-9qm4h"
Jul 27 20:19:35.484520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.483943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5d446c6b-9qm4h" event={"ID":"b9715601-a2cf-44eb-8eb4-862ccffde093","Type":"ContainerDied","Data":"4d027bddb26599a6ababfe7c75ead9e8b0e679f01702f0819e322a5c932b0991"}
Jul 27 20:19:35.484520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.484070    2560 scope.go:117] "RemoveContainer" containerID="3f3a0ebe14998aa4100e99e4ab538af84dd625f642d6f424d71aeac2420cb04b"
Jul 27 20:19:35.505281 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.505243    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:19:35.508316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.508290    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b5d446c6b-9qm4h"]
Jul 27 20:19:35.725318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:35.725278    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9715601-a2cf-44eb-8eb4-862ccffde093" path="/var/lib/kubelet/pods/b9715601-a2cf-44eb-8eb4-862ccffde093/volumes"
Jul 27 20:19:37.371699 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:37.371667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:19:37.375614 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:37.375594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b795b9688-fmvnd"
Jul 27 20:19:41.340353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:41.340313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7llv"
Jul 27 20:19:59.261347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:59.261304    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:19:59.317987 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:59.317957    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:19:59.605965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:19:59.605895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:13.090464 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090426    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090731    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="alertmanager" containerID="cri-o://97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3" gracePeriod=120
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090800    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-metric" containerID="cri-o://68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d" gracePeriod=120
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090863    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="prom-label-proxy" containerID="cri-o://27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e" gracePeriod=120
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090894    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-web" containerID="cri-o://b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047" gracePeriod=120
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090916    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy" containerID="cri-o://450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f" gracePeriod=120
Jul 27 20:20:13.091005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.090910    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="config-reloader" containerID="cri-o://2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36" gracePeriod=120
Jul 27 20:20:13.645348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645317    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e" exitCode=0
Jul 27 20:20:13.645348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645345    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d" exitCode=0
Jul 27 20:20:13.645348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645354    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f" exitCode=0
Jul 27 20:20:13.645348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645362    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36" exitCode=0
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645371    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3" exitCode=0
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"}
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"}
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"}
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645511    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"}
Jul 27 20:20:13.645629 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:13.645536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"}
Jul 27 20:20:14.333035 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.333012    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:20:14.430830 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430724    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.430830 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430902    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430927    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430959    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz6tf\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.430988    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431076    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431125    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431158    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.431484 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431423    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:14.431709 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:14.431862 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.431732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume\") pod \"3a84902a-4439-4e01-9552-475e1127a225\" (UID: \"3a84902a-4439-4e01-9552-475e1127a225\") "
Jul 27 20:20:14.432184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.432060    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-main-db\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.432184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.432086    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.432749 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.432708    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:14.434545 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.434516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:14.435001 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.434974    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out" (OuterVolumeSpecName: "config-out") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:14.435093 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume" (OuterVolumeSpecName: "config-volume") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.435372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.435454 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435388    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf" (OuterVolumeSpecName: "kube-api-access-qz6tf") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "kube-api-access-qz6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:14.435516 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435489    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.435601 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435582    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.435850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.435824    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.438578 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.438558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.445548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.445525    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config" (OuterVolumeSpecName: "web-config") pod "3a84902a-4439-4e01-9552-475e1127a225" (UID: "3a84902a-4439-4e01-9552-475e1127a225"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:14.533213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533161    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-tls-assets\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533204    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533215    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-web-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533226    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-main-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533235    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-cluster-tls-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533245    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6tf\" (UniqueName: \"kubernetes.io/projected/3a84902a-4439-4e01-9552-475e1127a225-kube-api-access-qz6tf\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533254    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533262    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a84902a-4439-4e01-9552-475e1127a225-metrics-client-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533270    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a84902a-4439-4e01-9552-475e1127a225-config-out\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533278    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-config-volume\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.533509 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.533285    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3a84902a-4439-4e01-9552-475e1127a225-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:14.651996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.651964    2560 generic.go:356] "Generic (PLEG): container finished" podID="3a84902a-4439-4e01-9552-475e1127a225" containerID="b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047" exitCode=0
Jul 27 20:20:14.652167 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.652042    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"}
Jul 27 20:20:14.652167 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.652057    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:20:14.652167 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.652139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3a84902a-4439-4e01-9552-475e1127a225","Type":"ContainerDied","Data":"8892995dc06bda0a259daece7dd37b99fafb27a365b190bf0a76785e27945a78"}
Jul 27 20:20:14.652267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.652175    2560 scope.go:117] "RemoveContainer" containerID="27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"
Jul 27 20:20:14.660273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.660250    2560 scope.go:117] "RemoveContainer" containerID="68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"
Jul 27 20:20:14.667481 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.667465    2560 scope.go:117] "RemoveContainer" containerID="450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"
Jul 27 20:20:14.674854 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.674762    2560 scope.go:117] "RemoveContainer" containerID="b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"
Jul 27 20:20:14.676512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.676487    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:20:14.682092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.682040    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:20:14.682883 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.682868    2560 scope.go:117] "RemoveContainer" containerID="2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"
Jul 27 20:20:14.689744 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.689729    2560 scope.go:117] "RemoveContainer" containerID="97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"
Jul 27 20:20:14.696680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.696663    2560 scope.go:117] "RemoveContainer" containerID="c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e"
Jul 27 20:20:14.704191 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704167    2560 scope.go:117] "RemoveContainer" containerID="27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"
Jul 27 20:20:14.704443 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.704425    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e\": container with ID starting with 27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e not found: ID does not exist" containerID="27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"
Jul 27 20:20:14.704492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704453    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e"} err="failed to get container status \"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e\": rpc error: code = NotFound desc = could not find container \"27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e\": container with ID starting with 27c86971f58fb774a0b77b7e51733cd2fc5ce6f9961b3a56bb11082d331c391e not found: ID does not exist"
Jul 27 20:20:14.704492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704486    2560 scope.go:117] "RemoveContainer" containerID="68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"
Jul 27 20:20:14.704708 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.704693    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d\": container with ID starting with 68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d not found: ID does not exist" containerID="68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"
Jul 27 20:20:14.704745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d"} err="failed to get container status \"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d\": rpc error: code = NotFound desc = could not find container \"68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d\": container with ID starting with 68ad0aad80a4858b6eddfbde6a3b679266e7dd029ab01aab31e597bf5201463d not found: ID does not exist"
Jul 27 20:20:14.704745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704724    2560 scope.go:117] "RemoveContainer" containerID="450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"
Jul 27 20:20:14.704988 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.704970    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f\": container with ID starting with 450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f not found: ID does not exist" containerID="450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"
Jul 27 20:20:14.705035 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.704992    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f"} err="failed to get container status \"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f\": rpc error: code = NotFound desc = could not find container \"450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f\": container with ID starting with 450dc59e085d11e0ee59c932b32e0c4de572331958251d9ce1f2a8eac124be2f not found: ID does not exist"
Jul 27 20:20:14.705035 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705010    2560 scope.go:117] "RemoveContainer" containerID="b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"
Jul 27 20:20:14.705217 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.705201    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047\": container with ID starting with b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047 not found: ID does not exist" containerID="b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"
Jul 27 20:20:14.705262 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705220    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047"} err="failed to get container status \"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047\": rpc error: code = NotFound desc = could not find container \"b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047\": container with ID starting with b887ee0f52bc75057f89fbc9083fe363f6ac6daddb6f91a7e953d0e345c51047 not found: ID does not exist"
Jul 27 20:20:14.705262 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705232    2560 scope.go:117] "RemoveContainer" containerID="2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"
Jul 27 20:20:14.705446 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.705424    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36\": container with ID starting with 2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36 not found: ID does not exist" containerID="2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"
Jul 27 20:20:14.705480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36"} err="failed to get container status \"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36\": rpc error: code = NotFound desc = could not find container \"2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36\": container with ID starting with 2af839edde9bdf1dbb781d0a805009f0a1fdb330c02267375bc528bb1d6cba36 not found: ID does not exist"
Jul 27 20:20:14.705480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705461    2560 scope.go:117] "RemoveContainer" containerID="97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"
Jul 27 20:20:14.705696 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.705679    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3\": container with ID starting with 97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3 not found: ID does not exist" containerID="97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"
Jul 27 20:20:14.705746 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705700    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3"} err="failed to get container status \"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3\": rpc error: code = NotFound desc = could not find container \"97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3\": container with ID starting with 97a0f56933b6237d3699d51300ec9eae7ba1ca08128e87dc88bb52edafe389e3 not found: ID does not exist"
Jul 27 20:20:14.705746 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705716    2560 scope.go:117] "RemoveContainer" containerID="c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e"
Jul 27 20:20:14.705956 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:20:14.705939    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e\": container with ID starting with c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e not found: ID does not exist" containerID="c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e"
Jul 27 20:20:14.706003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:14.705961    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e"} err="failed to get container status \"c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e\": rpc error: code = NotFound desc = could not find container \"c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e\": container with ID starting with c6e19fd01e1b32e2ad4ad4ad8e7debb046dc3cf776a455b3bb3921a401b47a6e not found: ID does not exist"
Jul 27 20:20:15.726553 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:15.726519    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a84902a-4439-4e01-9552-475e1127a225" path="/var/lib/kubelet/pods/3a84902a-4439-4e01-9552-475e1127a225/volumes"
Jul 27 20:20:17.327367 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.327332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:17.327799 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.327758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="prometheus" containerID="cri-o://e125b9e323204138cfe0ee4f42df1aef4646b07586ce78daaa8673c04ed9c710" gracePeriod=600
Jul 27 20:20:17.327887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.327789    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy" containerID="cri-o://c350465cc200318e143c45622a5085c057eb53b0d7ec5ed30985d2709e38681f" gracePeriod=600
Jul 27 20:20:17.327957 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.327934    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="config-reloader" containerID="cri-o://16daa68759c840266c7ecb3cec57cd3317a60db46ff797d2eafcf4fac3774807" gracePeriod=600
Jul 27 20:20:17.328020 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.327951    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2931a2706594021cd0af6693177a16592796dc873047b34b32ff408768af7281" gracePeriod=600
Jul 27 20:20:17.328072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.328030    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-web" containerID="cri-o://64459fe54f560bd449b47294be73175632807599339efd22433cfd570bdff333" gracePeriod=600
Jul 27 20:20:17.328122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.328101    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="thanos-sidecar" containerID="cri-o://f8e96a94d527d2370d57d7a1f46a41dcf91742ec9716961591e8f97167832bfc" gracePeriod=600
Jul 27 20:20:17.668364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668324    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="2931a2706594021cd0af6693177a16592796dc873047b34b32ff408768af7281" exitCode=0
Jul 27 20:20:17.668364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668355    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="c350465cc200318e143c45622a5085c057eb53b0d7ec5ed30985d2709e38681f" exitCode=0
Jul 27 20:20:17.668364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668363    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="64459fe54f560bd449b47294be73175632807599339efd22433cfd570bdff333" exitCode=0
Jul 27 20:20:17.668364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668369    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="f8e96a94d527d2370d57d7a1f46a41dcf91742ec9716961591e8f97167832bfc" exitCode=0
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668375    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="16daa68759c840266c7ecb3cec57cd3317a60db46ff797d2eafcf4fac3774807" exitCode=0
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668383    2560 generic.go:356] "Generic (PLEG): container finished" podID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerID="e125b9e323204138cfe0ee4f42df1aef4646b07586ce78daaa8673c04ed9c710" exitCode=0
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"2931a2706594021cd0af6693177a16592796dc873047b34b32ff408768af7281"}
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"c350465cc200318e143c45622a5085c057eb53b0d7ec5ed30985d2709e38681f"}
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"64459fe54f560bd449b47294be73175632807599339efd22433cfd570bdff333"}
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"f8e96a94d527d2370d57d7a1f46a41dcf91742ec9716961591e8f97167832bfc"}
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"16daa68759c840266c7ecb3cec57cd3317a60db46ff797d2eafcf4fac3774807"}
Jul 27 20:20:17.668646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.668632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"e125b9e323204138cfe0ee4f42df1aef4646b07586ce78daaa8673c04ed9c710"}
Jul 27 20:20:17.684171 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.684125    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:17.761114 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761114 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761116    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761225    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761355    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761462    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761545    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6kgf\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761578    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.761648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.762062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.762062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.762062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761739    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.762062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.761768    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0\") pod \"828a9a41-6c4e-4df1-b162-5f42071a4c82\" (UID: \"828a9a41-6c4e-4df1-b162-5f42071a4c82\") "
Jul 27 20:20:17.762302 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.762268    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.763839 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.763649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.764200 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.764098    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.764200 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.764154    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.764352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.764307    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.764666 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.764639    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.764994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.764968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:17.765299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.765261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:17.765505 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.765485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.765610 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.765569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.765746 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.765709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.765838 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.765792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.766113 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.766089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config" (OuterVolumeSpecName: "config") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.766850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.766742    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.766977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.766945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out" (OuterVolumeSpecName: "config-out") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:17.766977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.766971    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf" (OuterVolumeSpecName: "kube-api-access-x6kgf") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "kube-api-access-x6kgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:17.767099 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.766960    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.767398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.767377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:17.767733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.767713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.774866 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.774839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config" (OuterVolumeSpecName: "web-config") pod "828a9a41-6c4e-4df1-b162-5f42071a4c82" (UID: "828a9a41-6c4e-4df1-b162-5f42071a4c82"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862789    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-grpc-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862842    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-tls-assets\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862852    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862862    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862873    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862882    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862893    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-web-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.862896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862901    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-config\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862909    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862918    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-kube-rbac-proxy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862926    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862935    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862943    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/828a9a41-6c4e-4df1-b162-5f42071a4c82-secret-metrics-client-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862952    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-metrics-client-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862961    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862970    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862978    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862986    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6kgf\" (UniqueName: \"kubernetes.io/projected/828a9a41-6c4e-4df1-b162-5f42071a4c82-kube-api-access-x6kgf\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.862994    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-config-out\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:17.863235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:17.863002    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/828a9a41-6c4e-4df1-b162-5f42071a4c82-prometheus-k8s-db\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:20:18.675199 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.675163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"828a9a41-6c4e-4df1-b162-5f42071a4c82","Type":"ContainerDied","Data":"bc43bfcbbc3b70c63ae6535aa7a84224e83dfef56a493282bf6b334d1f643f0c"}
Jul 27 20:20:18.675199 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.675190    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.675764 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.675247    2560 scope.go:117] "RemoveContainer" containerID="2931a2706594021cd0af6693177a16592796dc873047b34b32ff408768af7281"
Jul 27 20:20:18.683482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.683372    2560 scope.go:117] "RemoveContainer" containerID="c350465cc200318e143c45622a5085c057eb53b0d7ec5ed30985d2709e38681f"
Jul 27 20:20:18.690270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.690251    2560 scope.go:117] "RemoveContainer" containerID="64459fe54f560bd449b47294be73175632807599339efd22433cfd570bdff333"
Jul 27 20:20:18.697118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.697102    2560 scope.go:117] "RemoveContainer" containerID="f8e96a94d527d2370d57d7a1f46a41dcf91742ec9716961591e8f97167832bfc"
Jul 27 20:20:18.700022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.700002    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:18.704733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.704710    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:18.706466 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.706451    2560 scope.go:117] "RemoveContainer" containerID="16daa68759c840266c7ecb3cec57cd3317a60db46ff797d2eafcf4fac3774807"
Jul 27 20:20:18.713577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.713560    2560 scope.go:117] "RemoveContainer" containerID="e125b9e323204138cfe0ee4f42df1aef4646b07586ce78daaa8673c04ed9c710"
Jul 27 20:20:18.720606 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.720590    2560 scope.go:117] "RemoveContainer" containerID="99356f9144a8f1ccc5dca32cf989284ce6428553f332d05b6d83443686a16106"
Jul 27 20:20:18.731645 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731625    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:18.731948 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731933    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="config-reloader"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731950    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="config-reloader"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731960    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731966    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731972    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9715601-a2cf-44eb-8eb4-862ccffde093" containerName="console"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731977    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9715601-a2cf-44eb-8eb4-862ccffde093" containerName="console"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731983    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="config-reloader"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731989    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="config-reloader"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.731996    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="prom-label-proxy"
Jul 27 20:20:18.731999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732002    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="prom-label-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732011    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-thanos"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732016    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-thanos"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732024    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="init-config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732029    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="init-config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732035    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732040    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732047    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="alertmanager"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732052    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="alertmanager"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732057    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732062    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732068    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="thanos-sidecar"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732072    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="thanos-sidecar"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732080    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="init-config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732085    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="init-config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732096    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="prometheus"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732100    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="prometheus"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732105    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732110    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732117    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-metric"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732123    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-metric"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732166    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9715601-a2cf-44eb-8eb4-862ccffde093" containerName="console"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732174    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732181    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732187    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="config-reloader"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732192    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="alertmanager"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732198    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="prometheus"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732204    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy-thanos"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732210    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="thanos-sidecar"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732216    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-web"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732220    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy-metric"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732225    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732230    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="kube-rbac-proxy"
Jul 27 20:20:18.732272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.732235    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84902a-4439-4e01-9552-475e1127a225" containerName="prom-label-proxy"
Jul 27 20:20:18.736156 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.736141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.738997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.738973    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dqc72skpp8c9n\""
Jul 27 20:20:18.739128 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.739106    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:20:18.739889 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.739858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:20:18.740373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.740345    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:20:18.740459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.740412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:20:18.740553 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.740538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:20:18.742747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.740863    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:20:18.742747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.741467    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:20:18.742747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.741656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:20:18.743458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.742826    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jvlxj\""
Jul 27 20:20:18.743458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.743102    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:20:18.743458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.743140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:20:18.743458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.743390    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:20:18.743717 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.743470    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:20:18.744781 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.744758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:20:18.747045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.747024    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:20:18.761670 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.761458    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:18.770121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x52kg\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-kube-api-access-x52kg\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.770748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.770547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x52kg\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-kube-api-access-x52kg\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.871788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.872235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.871904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.872235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.872085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.872943 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.872918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.873562 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.873531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.873662 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.873538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.873847 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.873823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.874036 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.874011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.874626 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.874602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.875133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.875089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.875233 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.875144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.876996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.876934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.876996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.876949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.876996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.876940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.877202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.877088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.877202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.877093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.877202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.877112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.877460 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.877421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.877460 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.877419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.878332 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.878304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:18.879488 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:18.879470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x52kg\" (UniqueName: \"kubernetes.io/projected/ff3c5491-7869-4b0e-a8c4-2d5aeec18282-kube-api-access-x52kg\") pod \"prometheus-k8s-0\" (UID: \"ff3c5491-7869-4b0e-a8c4-2d5aeec18282\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:19.049175 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.049085    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:20:19.189223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.189070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:20:19.191314 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:20:19.191285    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3c5491_7869_4b0e_a8c4_2d5aeec18282.slice/crio-e065fa116d26bbd3dc01ffa96c9980b0b1dbb53950ac90120c46ddac852cdc44 WatchSource:0}: Error finding container e065fa116d26bbd3dc01ffa96c9980b0b1dbb53950ac90120c46ddac852cdc44: Status 404 returned error can't find the container with id e065fa116d26bbd3dc01ffa96c9980b0b1dbb53950ac90120c46ddac852cdc44
Jul 27 20:20:19.680653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.680613    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff3c5491-7869-4b0e-a8c4-2d5aeec18282" containerID="57c1d5e4889ffea3066be3b55a5a0b3f510a656af03cbc182b8b26db8e231758" exitCode=0
Jul 27 20:20:19.681113 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.680696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerDied","Data":"57c1d5e4889ffea3066be3b55a5a0b3f510a656af03cbc182b8b26db8e231758"}
Jul 27 20:20:19.681113 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.680768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"e065fa116d26bbd3dc01ffa96c9980b0b1dbb53950ac90120c46ddac852cdc44"}
Jul 27 20:20:19.725781 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:19.725756    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828a9a41-6c4e-4df1-b162-5f42071a4c82" path="/var/lib/kubelet/pods/828a9a41-6c4e-4df1-b162-5f42071a4c82/volumes"
Jul 27 20:20:20.689879 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.689843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"4d5b5ce70d3c266d26779d105057b6cbc082cb7040aef5c2f906634d7a5c0a12"}
Jul 27 20:20:20.690316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.689911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"f6999032b9cf37a85635b022dee9351d2f0c28f1ae97b31bcb8a67f4e701b412"}
Jul 27 20:20:20.690316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.689940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"2bce73d950312638cf063d00733295993d4f815dd5f978b269b9d7b07bd90bb8"}
Jul 27 20:20:20.690316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.689966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"ceca1a98db47676ebf695aedf8153b80b396f26eece726a9ccdb93be2064c487"}
Jul 27 20:20:20.690316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.689990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"e9f04b3a0c15769a4c220ab7277fec9c4fb333d36b70c3ea8b5462363d84d89a"}
Jul 27 20:20:20.690316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.690013    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff3c5491-7869-4b0e-a8c4-2d5aeec18282","Type":"ContainerStarted","Data":"3c3606b00b510a5c206152cfc71bbfadb8260b6b014fbecfa61c5d4d67eff129"}
Jul 27 20:20:20.719472 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:20.719372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.699728234 podStartE2EDuration="2.719328329s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.600095ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:20.716293699 +0000 UTC m=+153.566060527" watchObservedRunningTime="2026-07-27 20:20:20.719328329 +0000 UTC m=+153.569095153"
Jul 27 20:20:24.049978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:20:24.049944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:21:19.050239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:19.050200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:21:19.066151 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:19.066122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:21:19.931730 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:19.931700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:21:50.782926 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.782880    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"]
Jul 27 20:21:50.785120 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.785098    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.787852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.787828    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 20:21:50.787951 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.787866    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 20:21:50.787951 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.787879    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4f5tt\""
Jul 27 20:21:50.795056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.795036    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"]
Jul 27 20:21:50.848721 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.848683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qbt\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-kube-api-access-p2qbt\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.848890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.848725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.950081 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.950049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qbt\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-kube-api-access-p2qbt\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.950272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.950115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.964942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.961897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:50.964942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:50.962341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qbt\" (UniqueName: \"kubernetes.io/projected/ef990db9-fe09-4e67-a999-1ce7c1f2f5e7-kube-api-access-p2qbt\") pod \"cert-manager-cainjector-559cdd8c87-pd7x5\" (UID: \"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:51.108805 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:51.108704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"
Jul 27 20:21:51.235209 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:51.235181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5"]
Jul 27 20:21:51.237957 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:21:51.237923    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef990db9_fe09_4e67_a999_1ce7c1f2f5e7.slice/crio-63500985555eb05d139c887a69cdf602e09b3079e0c9832c35044b3f63e7ff97 WatchSource:0}: Error finding container 63500985555eb05d139c887a69cdf602e09b3079e0c9832c35044b3f63e7ff97: Status 404 returned error can't find the container with id 63500985555eb05d139c887a69cdf602e09b3079e0c9832c35044b3f63e7ff97
Jul 27 20:21:52.037214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:52.037177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5" event={"ID":"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7","Type":"ContainerStarted","Data":"63500985555eb05d139c887a69cdf602e09b3079e0c9832c35044b3f63e7ff97"}
Jul 27 20:21:55.052594 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:55.052559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5" event={"ID":"ef990db9-fe09-4e67-a999-1ce7c1f2f5e7","Type":"ContainerStarted","Data":"b6a933f57218cf241a3ab29c75f09b8b1c67e66b0a4f88af55197fb51e4764c5"}
Jul 27 20:21:55.069682 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:21:55.069636    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pd7x5" podStartSLOduration=2.089189685 podStartE2EDuration="5.06961839s" totalImagesPullingTime="2.980428705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:21:55.068318462 +0000 UTC m=+247.918085294" watchObservedRunningTime="2026-07-27 20:21:55.06961839 +0000 UTC m=+247.919385221"
Jul 27 20:22:03.713409 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.713367    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"]
Jul 27 20:22:03.715705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.715687    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.718210 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.718182    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 20:22:03.718324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.718275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-h4frz\""
Jul 27 20:22:03.718324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.718280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:22:03.728796 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.728772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"]
Jul 27 20:22:03.742886 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.742858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.743041 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.742897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfbj\" (UniqueName: \"kubernetes.io/projected/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-kube-api-access-gcfbj\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.844180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.844138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.844180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.844180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfbj\" (UniqueName: \"kubernetes.io/projected/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-kube-api-access-gcfbj\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.844537 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.844517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:03.852939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:03.852905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfbj\" (UniqueName: \"kubernetes.io/projected/027b3c70-f1cd-4527-8ad6-32a4f6a6094b-kube-api-access-gcfbj\") pod \"openshift-lws-operator-fd8ccff4c-hjxmh\" (UID: \"027b3c70-f1cd-4527-8ad6-32a4f6a6094b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:04.025565 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:04.025486    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"
Jul 27 20:22:04.153020 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:04.152994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh"]
Jul 27 20:22:04.155045 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:22:04.155017    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027b3c70_f1cd_4527_8ad6_32a4f6a6094b.slice/crio-ce060790b9f1867b4ee477fd90be67ae7151b0180278e49c1beba814620bd59d WatchSource:0}: Error finding container ce060790b9f1867b4ee477fd90be67ae7151b0180278e49c1beba814620bd59d: Status 404 returned error can't find the container with id ce060790b9f1867b4ee477fd90be67ae7151b0180278e49c1beba814620bd59d
Jul 27 20:22:05.093176 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:05.093134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh" event={"ID":"027b3c70-f1cd-4527-8ad6-32a4f6a6094b","Type":"ContainerStarted","Data":"ce060790b9f1867b4ee477fd90be67ae7151b0180278e49c1beba814620bd59d"}
Jul 27 20:22:07.103282 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:07.103251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh" event={"ID":"027b3c70-f1cd-4527-8ad6-32a4f6a6094b","Type":"ContainerStarted","Data":"9f44c33d25c566e7a4de8a165fd2175c15eb36a7ab2eb32cbb32395d945c7361"}
Jul 27 20:22:07.121221 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:07.121163    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hjxmh" podStartSLOduration=1.747268578 podStartE2EDuration="4.121146542s" totalImagesPullingTime="2.373877964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:07.119267159 +0000 UTC m=+259.969033989" watchObservedRunningTime="2026-07-27 20:22:07.121146542 +0000 UTC m=+259.970913373"
Jul 27 20:22:22.694299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.694225    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"]
Jul 27 20:22:22.700693 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.700673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.703528 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.703507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:22:22.705228 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.705211    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jw6l6\""
Jul 27 20:22:22.705370 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.705338    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:22:22.705479 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.705426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:22:22.705965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.705944    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:22:22.729110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.729075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"]
Jul 27 20:22:22.819435 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.819399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.819435 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.819439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-manifests\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.819654 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.819533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.819740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.819721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdkk\" (UniqueName: \"kubernetes.io/projected/07bde055-9699-4385-9b10-4974e577e439-kube-api-access-pbdkk\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.819939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.819919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-tmp\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-tmp\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-manifests\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdkk\" (UniqueName: \"kubernetes.io/projected/07bde055-9699-4385-9b10-4974e577e439-kube-api-access-pbdkk\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.920897 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.920872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-manifests\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.922718 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.922692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07bde055-9699-4385-9b10-4974e577e439-tmp\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.922835 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.922800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.922931 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.922915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07bde055-9699-4385-9b10-4974e577e439-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:22.932166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:22.932144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdkk\" (UniqueName: \"kubernetes.io/projected/07bde055-9699-4385-9b10-4974e577e439-kube-api-access-pbdkk\") pod \"opendatahub-operator-controller-manager-6b8fd6b656-22wbs\" (UID: \"07bde055-9699-4385-9b10-4974e577e439\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:23.010519 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:23.010427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:23.149802 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:23.149778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"]
Jul 27 20:22:23.152310 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:22:23.152283    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bde055_9699_4385_9b10_4974e577e439.slice/crio-55ab7c4fa04cc70ec46568185b450a768cdabfd6ebfdb2cd53004e7f5a87caf6 WatchSource:0}: Error finding container 55ab7c4fa04cc70ec46568185b450a768cdabfd6ebfdb2cd53004e7f5a87caf6: Status 404 returned error can't find the container with id 55ab7c4fa04cc70ec46568185b450a768cdabfd6ebfdb2cd53004e7f5a87caf6
Jul 27 20:22:23.173162 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:23.173125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs" event={"ID":"07bde055-9699-4385-9b10-4974e577e439","Type":"ContainerStarted","Data":"55ab7c4fa04cc70ec46568185b450a768cdabfd6ebfdb2cd53004e7f5a87caf6"}
Jul 27 20:22:27.197455 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:27.197398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs" event={"ID":"07bde055-9699-4385-9b10-4974e577e439","Type":"ContainerStarted","Data":"c3eb4ced307a4492bf5a87d8a24f066808d9b918690b74f7edf3609fb11079a6"}
Jul 27 20:22:28.203156 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:28.203117    2560 generic.go:356] "Generic (PLEG): container finished" podID="07bde055-9699-4385-9b10-4974e577e439" containerID="c3eb4ced307a4492bf5a87d8a24f066808d9b918690b74f7edf3609fb11079a6" exitCode=0
Jul 27 20:22:28.203543 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:28.203207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs" event={"ID":"07bde055-9699-4385-9b10-4974e577e439","Type":"ContainerDied","Data":"c3eb4ced307a4492bf5a87d8a24f066808d9b918690b74f7edf3609fb11079a6"}
Jul 27 20:22:29.209267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:29.209231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs" event={"ID":"07bde055-9699-4385-9b10-4974e577e439","Type":"ContainerStarted","Data":"3562a4a14a90f79fbdf78739694516c5363d9b88055a076066d0fa4bf28314b0"}
Jul 27 20:22:29.209758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:29.209318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:29.232766 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:29.232716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs" podStartSLOduration=2.199979419 podStartE2EDuration="7.232701897s" totalImagesPullingTime="3.93401341s" totalInitContainerRuntime="1.098709068s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:29.232520327 +0000 UTC m=+282.082287157" watchObservedRunningTime="2026-07-27 20:22:29.232701897 +0000 UTC m=+282.082468724"
Jul 27 20:22:40.216081 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:40.216050    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8fd6b656-22wbs"
Jul 27 20:22:47.555715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:47.555688    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:22:48.198905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.198866    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"]
Jul 27 20:22:48.203553 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.203523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.206461 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.206427    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:22:48.206461 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.206455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:22:48.206707 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.206437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:22:48.206848 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.206832    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:22:48.207778 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.207762    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-msvdv\""
Jul 27 20:22:48.210775 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.210752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"]
Jul 27 20:22:48.247955 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.247910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttgq\" (UniqueName: \"kubernetes.io/projected/6b0807fb-7a28-4976-8fbf-126a94f42d58-kube-api-access-tttgq\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.248138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.247990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b0807fb-7a28-4976-8fbf-126a94f42d58-tmp\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.248138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.248016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.348975 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.348938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttgq\" (UniqueName: \"kubernetes.io/projected/6b0807fb-7a28-4976-8fbf-126a94f42d58-kube-api-access-tttgq\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.349180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.349011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b0807fb-7a28-4976-8fbf-126a94f42d58-tmp\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.349180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.349042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.349180 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:22:48.349135    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 20:22:48.349335 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:22:48.349211    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs podName:6b0807fb-7a28-4976-8fbf-126a94f42d58 nodeName:}" failed. No retries permitted until 2026-07-27 20:22:48.849188919 +0000 UTC m=+301.698955740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs") pod "kube-auth-proxy-59b675654c-xvfkk" (UID: "6b0807fb-7a28-4976-8fbf-126a94f42d58") : secret "kube-auth-proxy-tls" not found
Jul 27 20:22:48.351354 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.351329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b0807fb-7a28-4976-8fbf-126a94f42d58-tmp\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.360121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.360092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttgq\" (UniqueName: \"kubernetes.io/projected/6b0807fb-7a28-4976-8fbf-126a94f42d58-kube-api-access-tttgq\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.854203 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:48.854161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:48.854623 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:22:48.854317    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 20:22:48.854623 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:22:48.854397    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs podName:6b0807fb-7a28-4976-8fbf-126a94f42d58 nodeName:}" failed. No retries permitted until 2026-07-27 20:22:49.854379557 +0000 UTC m=+302.704146369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs") pod "kube-auth-proxy-59b675654c-xvfkk" (UID: "6b0807fb-7a28-4976-8fbf-126a94f42d58") : secret "kube-auth-proxy-tls" not found
Jul 27 20:22:49.863829 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:49.863778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:49.866155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:49.866132    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0807fb-7a28-4976-8fbf-126a94f42d58-tls-certs\") pod \"kube-auth-proxy-59b675654c-xvfkk\" (UID: \"6b0807fb-7a28-4976-8fbf-126a94f42d58\") " pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:50.015332 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:50.015293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"
Jul 27 20:22:50.149584 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:50.149547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b675654c-xvfkk"]
Jul 27 20:22:50.152485 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:22:50.152463    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0807fb_7a28_4976_8fbf_126a94f42d58.slice/crio-8dfa3d91f5eb23db4eb61ca7ff91920e1c0664996f07a89dfa26d4fdad21b342 WatchSource:0}: Error finding container 8dfa3d91f5eb23db4eb61ca7ff91920e1c0664996f07a89dfa26d4fdad21b342: Status 404 returned error can't find the container with id 8dfa3d91f5eb23db4eb61ca7ff91920e1c0664996f07a89dfa26d4fdad21b342
Jul 27 20:22:50.155008 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:50.154992    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:22:50.302276 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:50.302243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk" event={"ID":"6b0807fb-7a28-4976-8fbf-126a94f42d58","Type":"ContainerStarted","Data":"8dfa3d91f5eb23db4eb61ca7ff91920e1c0664996f07a89dfa26d4fdad21b342"}
Jul 27 20:22:51.531353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.530422    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"]
Jul 27 20:22:51.534517 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.534488    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.537311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.537117    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 20:22:51.537311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.537298    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 20:22:51.538473 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.538453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 20:22:51.538598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.538486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xd2vh\""
Jul 27 20:22:51.547450 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.547418    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"]
Jul 27 20:22:51.586115 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.586075    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-metrics-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.586307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.586167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-kube-api-access-shv24\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.586307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.586222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.586429 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.586328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-manager-config\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.687502 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.687460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-metrics-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.687701 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.687528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-kube-api-access-shv24\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.687701 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.687572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.687701 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.687658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-manager-config\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.688578 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.688530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-manager-config\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.690474 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.690444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-metrics-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.691652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.691624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-cert\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.703044 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.703008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv24\" (UniqueName: \"kubernetes.io/projected/bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5-kube-api-access-shv24\") pod \"lws-controller-manager-57bc674ff4-bwkqq\" (UID: \"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5\") " pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:51.850245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:51.850154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:52.662355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:52.662332    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"]
Jul 27 20:22:52.664759 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:22:52.664736    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39830f_b6eb_4c8a_8c1a_57efca5bb4d5.slice/crio-f03ecb82128f4732b0f67ae1cc7126dcf8a66723aee544a8f898408190fe8f0a WatchSource:0}: Error finding container f03ecb82128f4732b0f67ae1cc7126dcf8a66723aee544a8f898408190fe8f0a: Status 404 returned error can't find the container with id f03ecb82128f4732b0f67ae1cc7126dcf8a66723aee544a8f898408190fe8f0a
Jul 27 20:22:53.319591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:53.319555    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk" event={"ID":"6b0807fb-7a28-4976-8fbf-126a94f42d58","Type":"ContainerStarted","Data":"63559c665c15ba75632e5ceb59149c4264dfce07661dc20389aba26de30213dd"}
Jul 27 20:22:53.320836 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:53.320797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq" event={"ID":"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5","Type":"ContainerStarted","Data":"f03ecb82128f4732b0f67ae1cc7126dcf8a66723aee544a8f898408190fe8f0a"}
Jul 27 20:22:53.339386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:53.339328    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59b675654c-xvfkk" podStartSLOduration=2.960156159 podStartE2EDuration="5.339310308s" totalImagesPullingTime="2.379154149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:53.338129748 +0000 UTC m=+306.187896578" watchObservedRunningTime="2026-07-27 20:22:53.339310308 +0000 UTC m=+306.189077137"
Jul 27 20:22:55.332045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:55.332004    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq" event={"ID":"bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5","Type":"ContainerStarted","Data":"98c69f9a1a9b7b94974df42ee61f7189167d9c645d1983b29c1666d72634b1d1"}
Jul 27 20:22:55.332501 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:55.332081    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:22:55.348577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:22:55.348518    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq" podStartSLOduration=2.22621364 podStartE2EDuration="4.348501171s" totalImagesPullingTime="2.122287531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:55.347512843 +0000 UTC m=+308.197279674" watchObservedRunningTime="2026-07-27 20:22:55.348501171 +0000 UTC m=+308.198268000"
Jul 27 20:23:06.338558 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:23:06.338527    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57bc674ff4-bwkqq"
Jul 27 20:24:46.231940 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.231908    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"]
Jul 27 20:24:46.234985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.234966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.237659 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.237632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-55ddz\""
Jul 27 20:24:46.237784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.237661    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:24:46.237944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.237927    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:24:46.250155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.250132    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"]
Jul 27 20:24:46.341579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.341548    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"]
Jul 27 20:24:46.341840 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:46.341796    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2ffwg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262"
Jul 27 20:24:46.397338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.397303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffwg\" (UniqueName: \"kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.397487 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.397357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.498387 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.498316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.498387 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.498383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffwg\" (UniqueName: \"kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.498679 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.498660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.507235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.507212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffwg\" (UniqueName: \"kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.818913 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.818837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.825788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.825758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.828061 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.828036    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"]
Jul 27 20:24:46.833259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.833235    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"]
Jul 27 20:24:46.837010 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.836988    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:24:46.838708 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:46.838672    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6625add1-9a70-407c-af6b-cb2c4f84c262\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\": pods \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.839983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.839968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:46.841287 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:46.841263    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" is forbidden: User \"system:node:ip-10-0-143-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-74.ec2.internal' and this object" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.843513 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:46.843490    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" is forbidden: User \"system:node:ip-10-0-143-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-74.ec2.internal' and this object" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:46.855909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.855885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:24:46.901050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.901017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ffwg\" (UniqueName: \"kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg\") pod \"6625add1-9a70-407c-af6b-cb2c4f84c262\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") "
Jul 27 20:24:46.901202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.901063    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume\") pod \"6625add1-9a70-407c-af6b-cb2c4f84c262\" (UID: \"6625add1-9a70-407c-af6b-cb2c4f84c262\") "
Jul 27 20:24:46.901367 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.901340    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6625add1-9a70-407c-af6b-cb2c4f84c262" (UID: "6625add1-9a70-407c-af6b-cb2c4f84c262"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:24:46.903058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:46.903030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg" (OuterVolumeSpecName: "kube-api-access-2ffwg") pod "6625add1-9a70-407c-af6b-cb2c4f84c262" (UID: "6625add1-9a70-407c-af6b-cb2c4f84c262"). InnerVolumeSpecName "kube-api-access-2ffwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:24:47.001733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.001697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmczv\" (UniqueName: \"kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.001892 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.001845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.001947 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.001896    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ffwg\" (UniqueName: \"kubernetes.io/projected/6625add1-9a70-407c-af6b-cb2c4f84c262-kube-api-access-2ffwg\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:24:47.001947 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.001907    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6625add1-9a70-407c-af6b-cb2c4f84c262-extensions-socket-volume\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:24:47.102414 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.102340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.102414 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.102398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmczv\" (UniqueName: \"kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.102704 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.102684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.115626 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.115601    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmczv\" (UniqueName: \"kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv\") pod \"kuadrant-operator-controller-manager-55467d9f7-crch5\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.150572 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.150547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:47.282398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.282374    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:24:47.285512 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:24:47.285480    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55628b55_d78d_42b2_bf87_d3ebff2ada3d.slice/crio-ab8dae6d3db1ac9de5ad7df1f361822e3574ed175ccd41a8a55e0d6a724a9c83 WatchSource:0}: Error finding container ab8dae6d3db1ac9de5ad7df1f361822e3574ed175ccd41a8a55e0d6a724a9c83: Status 404 returned error can't find the container with id ab8dae6d3db1ac9de5ad7df1f361822e3574ed175ccd41a8a55e0d6a724a9c83
Jul 27 20:24:47.725719 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:47.725681    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" is forbidden: User \"system:node:ip-10-0-143-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-74.ec2.internal' and this object" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:47.726473 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.726454    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262" path="/var/lib/kubelet/pods/6625add1-9a70-407c-af6b-cb2c4f84c262/volumes"
Jul 27 20:24:47.824994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.824962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" event={"ID":"55628b55-d78d-42b2-bf87-d3ebff2ada3d","Type":"ContainerStarted","Data":"ab8dae6d3db1ac9de5ad7df1f361822e3574ed175ccd41a8a55e0d6a724a9c83"}
Jul 27 20:24:47.824994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:47.824995    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:47.829502 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:24:47.829469    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-lsvhc\" is forbidden: User \"system:node:ip-10-0-143-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-74.ec2.internal' and this object" podUID="6625add1-9a70-407c-af6b-cb2c4f84c262" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lsvhc"
Jul 27 20:24:51.845207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:51.845176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" event={"ID":"55628b55-d78d-42b2-bf87-d3ebff2ada3d","Type":"ContainerStarted","Data":"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"}
Jul 27 20:24:51.845698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:51.845317    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:24:51.865745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:24:51.865685    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" podStartSLOduration=1.380456502 podStartE2EDuration="5.865668337s" totalImagesPullingTime="4.485211835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:24:51.863975974 +0000 UTC m=+424.713742806" watchObservedRunningTime="2026-07-27 20:24:51.865668337 +0000 UTC m=+424.715435168"
Jul 27 20:25:02.852219 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:02.852183    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:25:03.149566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.149525    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:25:03.153240 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.153222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.196636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.196597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:25:03.244199 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.244168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676jz\" (UniqueName: \"kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.244364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.244227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.344970 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.344929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.345171 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.345002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676jz\" (UniqueName: \"kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.345335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.345312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.354144 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.354108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676jz\" (UniqueName: \"kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz\") pod \"kuadrant-operator-controller-manager-86878c769f-z9k8l\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.463725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.463677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.595269 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.595236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:25:03.599475 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:25:03.599446    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdc25fa_2db2_4573_9e12_cf91ae3fab9a.slice/crio-d5bc91ad9f5ee410dc4ff7b38011e9920624c82227a86e9a19eb628f3aa1f597 WatchSource:0}: Error finding container d5bc91ad9f5ee410dc4ff7b38011e9920624c82227a86e9a19eb628f3aa1f597: Status 404 returned error can't find the container with id d5bc91ad9f5ee410dc4ff7b38011e9920624c82227a86e9a19eb628f3aa1f597
Jul 27 20:25:03.896583 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.896543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" event={"ID":"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a","Type":"ContainerStarted","Data":"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"}
Jul 27 20:25:03.897027 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.896602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" event={"ID":"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a","Type":"ContainerStarted","Data":"d5bc91ad9f5ee410dc4ff7b38011e9920624c82227a86e9a19eb628f3aa1f597"}
Jul 27 20:25:03.897027 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.896681    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:03.915671 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:03.915625    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" podStartSLOduration=0.915610952 podStartE2EDuration="915.610952ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:25:03.915074568 +0000 UTC m=+436.764841401" watchObservedRunningTime="2026-07-27 20:25:03.915610952 +0000 UTC m=+436.765377783"
Jul 27 20:25:14.903296 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:14.903229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:25:14.951343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:14.951310    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:25:14.951523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:14.951487    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" podUID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" containerName="manager" containerID="cri-o://23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b" gracePeriod=10
Jul 27 20:25:15.188627 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.188603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:25:15.238285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.238249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmczv\" (UniqueName: \"kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv\") pod \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") "
Jul 27 20:25:15.238285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.238290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume\") pod \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\" (UID: \"55628b55-d78d-42b2-bf87-d3ebff2ada3d\") "
Jul 27 20:25:15.238820 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.238786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "55628b55-d78d-42b2-bf87-d3ebff2ada3d" (UID: "55628b55-d78d-42b2-bf87-d3ebff2ada3d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:25:15.240385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.240356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv" (OuterVolumeSpecName: "kube-api-access-cmczv") pod "55628b55-d78d-42b2-bf87-d3ebff2ada3d" (UID: "55628b55-d78d-42b2-bf87-d3ebff2ada3d"). InnerVolumeSpecName "kube-api-access-cmczv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:25:15.339002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.338976    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmczv\" (UniqueName: \"kubernetes.io/projected/55628b55-d78d-42b2-bf87-d3ebff2ada3d-kube-api-access-cmczv\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:25:15.339002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.338997    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55628b55-d78d-42b2-bf87-d3ebff2ada3d-extensions-socket-volume\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:25:15.949401 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.949367    2560 generic.go:356] "Generic (PLEG): container finished" podID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" containerID="23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b" exitCode=0
Jul 27 20:25:15.949842 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.949425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" event={"ID":"55628b55-d78d-42b2-bf87-d3ebff2ada3d","Type":"ContainerDied","Data":"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"}
Jul 27 20:25:15.949842 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.949430    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"
Jul 27 20:25:15.949842 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.949468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5" event={"ID":"55628b55-d78d-42b2-bf87-d3ebff2ada3d","Type":"ContainerDied","Data":"ab8dae6d3db1ac9de5ad7df1f361822e3574ed175ccd41a8a55e0d6a724a9c83"}
Jul 27 20:25:15.949842 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.949489    2560 scope.go:117] "RemoveContainer" containerID="23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"
Jul 27 20:25:15.957971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.957747    2560 scope.go:117] "RemoveContainer" containerID="23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"
Jul 27 20:25:15.958095 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:25:15.958075    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b\": container with ID starting with 23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b not found: ID does not exist" containerID="23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"
Jul 27 20:25:15.958141 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.958102    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b"} err="failed to get container status \"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b\": rpc error: code = NotFound desc = could not find container \"23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b\": container with ID starting with 23b305575fdd731f2954fd8ac2f8cb093321a6bf1bfdd7d18e3733f0f6f1a11b not found: ID does not exist"
Jul 27 20:25:15.969177 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.969152    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:25:15.972464 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:15.972445    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-crch5"]
Jul 27 20:25:17.725597 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:17.725566    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" path="/var/lib/kubelet/pods/55628b55-d78d-42b2-bf87-d3ebff2ada3d/volumes"
Jul 27 20:25:34.126355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.126315    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:34.126965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.126827    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" containerName="manager"
Jul 27 20:25:34.126965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.126848    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" containerName="manager"
Jul 27 20:25:34.126965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.126951    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="55628b55-d78d-42b2-bf87-d3ebff2ada3d" containerName="manager"
Jul 27 20:25:34.131024 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.131002    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:34.133526 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.133506    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-glsmt\""
Jul 27 20:25:34.136128 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.136108    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:34.187433 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.187405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccsp\" (UniqueName: \"kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp\") pod \"authorino-58cd955675-fspjp\" (UID: \"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd\") " pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:34.288794 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.288764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccsp\" (UniqueName: \"kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp\") pod \"authorino-58cd955675-fspjp\" (UID: \"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd\") " pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:34.296778 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.296757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccsp\" (UniqueName: \"kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp\") pod \"authorino-58cd955675-fspjp\" (UID: \"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd\") " pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:34.441556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.441471    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:34.568233 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:34.568209    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:34.570986 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:25:34.570954    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7443a7d_1f71_4d3e_8f5d_f3b5220bc3cd.slice/crio-107062d1b97e1b80c6598a012f5b1b0f69dbc92be03405037b6f2f783ebbfce3 WatchSource:0}: Error finding container 107062d1b97e1b80c6598a012f5b1b0f69dbc92be03405037b6f2f783ebbfce3: Status 404 returned error can't find the container with id 107062d1b97e1b80c6598a012f5b1b0f69dbc92be03405037b6f2f783ebbfce3
Jul 27 20:25:35.031322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:35.031283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fspjp" event={"ID":"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd","Type":"ContainerStarted","Data":"107062d1b97e1b80c6598a012f5b1b0f69dbc92be03405037b6f2f783ebbfce3"}
Jul 27 20:25:38.047310 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:38.047268    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fspjp" event={"ID":"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd","Type":"ContainerStarted","Data":"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"}
Jul 27 20:25:38.062320 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:38.062273    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-fspjp" podStartSLOduration=0.749860058 podStartE2EDuration="4.062258523s" totalImagesPullingTime="3.312398465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:25:38.061680803 +0000 UTC m=+470.911447635" watchObservedRunningTime="2026-07-27 20:25:38.062258523 +0000 UTC m=+470.912025354"
Jul 27 20:25:39.643982 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:39.643948    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:40.056827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:40.056722    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-fspjp" podUID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" containerName="authorino" containerID="cri-o://aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5" gracePeriod=30
Jul 27 20:25:40.295167 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:40.295141    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:40.450383 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:40.450349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rccsp\" (UniqueName: \"kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp\") pod \"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd\" (UID: \"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd\") "
Jul 27 20:25:40.452390 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:40.452365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp" (OuterVolumeSpecName: "kube-api-access-rccsp") pod "e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" (UID: "e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd"). InnerVolumeSpecName "kube-api-access-rccsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:25:40.551225 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:40.551190    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccsp\" (UniqueName: \"kubernetes.io/projected/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd-kube-api-access-rccsp\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:25:41.062458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.062423    2560 generic.go:356] "Generic (PLEG): container finished" podID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" containerID="aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5" exitCode=0
Jul 27 20:25:41.062953 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.062466    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fspjp"
Jul 27 20:25:41.062953 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.062503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fspjp" event={"ID":"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd","Type":"ContainerDied","Data":"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"}
Jul 27 20:25:41.062953 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.062544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fspjp" event={"ID":"e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd","Type":"ContainerDied","Data":"107062d1b97e1b80c6598a012f5b1b0f69dbc92be03405037b6f2f783ebbfce3"}
Jul 27 20:25:41.062953 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.062575    2560 scope.go:117] "RemoveContainer" containerID="aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"
Jul 27 20:25:41.071251 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.071235    2560 scope.go:117] "RemoveContainer" containerID="aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"
Jul 27 20:25:41.071483 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:25:41.071465    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5\": container with ID starting with aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5 not found: ID does not exist" containerID="aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"
Jul 27 20:25:41.071526 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.071493    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5"} err="failed to get container status \"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5\": rpc error: code = NotFound desc = could not find container \"aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5\": container with ID starting with aa876b4868ea9b96c6e95905802bb9b96cde69e7ce025a4350d3b05e4d6ce5f5 not found: ID does not exist"
Jul 27 20:25:41.082975 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.082951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:41.086958 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.086938    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-fspjp"]
Jul 27 20:25:41.725749 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:25:41.725715    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" path="/var/lib/kubelet/pods/e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd/volumes"
Jul 27 20:26:05.336384 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.336346    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:05.336850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.336705    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" containerName="authorino"
Jul 27 20:26:05.336850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.336717    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" containerName="authorino"
Jul 27 20:26:05.336850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.336780    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7443a7d-1f71-4d3e-8f5d-f3b5220bc3cd" containerName="authorino"
Jul 27 20:26:05.341184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.341167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:05.344041 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.344020    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-glsmt\""
Jul 27 20:26:05.347793 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.347773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:05.359187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.359153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfv46\" (UniqueName: \"kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46\") pod \"authorino-5d4b67c446-twglv\" (UID: \"303a6c33-839a-4d09-b7ee-75ad2a82ce83\") " pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:05.460621 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.460590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfv46\" (UniqueName: \"kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46\") pod \"authorino-5d4b67c446-twglv\" (UID: \"303a6c33-839a-4d09-b7ee-75ad2a82ce83\") " pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:05.469707 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.469675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfv46\" (UniqueName: \"kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46\") pod \"authorino-5d4b67c446-twglv\" (UID: \"303a6c33-839a-4d09-b7ee-75ad2a82ce83\") " pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:05.575775 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.575741    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:05.575977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.575965    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:05.605235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.605205    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:05.609331 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.609307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:05.616873 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.616729    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:05.662342 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.662308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6c5v\" (UniqueName: \"kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v\") pod \"authorino-6986bb9f4d-9l9mt\" (UID: \"da21c4ab-2fe0-4927-ac92-87f8817cae05\") " pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:05.709957 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.709929    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:05.713293 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:26:05.713265    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303a6c33_839a_4d09_b7ee_75ad2a82ce83.slice/crio-1d3494aa0bd7b26cbd1e49bd6a51c920184696d8c8d75847b0a5e7e124fbd9da WatchSource:0}: Error finding container 1d3494aa0bd7b26cbd1e49bd6a51c920184696d8c8d75847b0a5e7e124fbd9da: Status 404 returned error can't find the container with id 1d3494aa0bd7b26cbd1e49bd6a51c920184696d8c8d75847b0a5e7e124fbd9da
Jul 27 20:26:05.763501 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.763479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6c5v\" (UniqueName: \"kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v\") pod \"authorino-6986bb9f4d-9l9mt\" (UID: \"da21c4ab-2fe0-4927-ac92-87f8817cae05\") " pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:05.771655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.771625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6c5v\" (UniqueName: \"kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v\") pod \"authorino-6986bb9f4d-9l9mt\" (UID: \"da21c4ab-2fe0-4927-ac92-87f8817cae05\") " pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:05.819591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.819563    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:05.819832 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.819800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:05.846251 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.846191    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:26:05.850607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.850587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.853304 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.853284    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 20:26:05.858977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.858955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:26:05.864191 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.864167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wqx\" (UniqueName: \"kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.864293 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.864200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.957016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.956989    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:05.960342 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:26:05.960313    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda21c4ab_2fe0_4927_ac92_87f8817cae05.slice/crio-484fd50d0a8451dac6d8433ca3d1959610b66bc1425779584824de484c99363a WatchSource:0}: Error finding container 484fd50d0a8451dac6d8433ca3d1959610b66bc1425779584824de484c99363a: Status 404 returned error can't find the container with id 484fd50d0a8451dac6d8433ca3d1959610b66bc1425779584824de484c99363a
Jul 27 20:26:05.965715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.965689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wqx\" (UniqueName: \"kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.965836 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.965724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.968127 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.968106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:05.973548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:05.973524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wqx\" (UniqueName: \"kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx\") pod \"authorino-d695cb8d8-vn5cz\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") " pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:06.161171 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.161149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:26:06.173563 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.173528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-twglv" event={"ID":"303a6c33-839a-4d09-b7ee-75ad2a82ce83","Type":"ContainerStarted","Data":"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"}
Jul 27 20:26:06.173716 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.173582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-twglv" event={"ID":"303a6c33-839a-4d09-b7ee-75ad2a82ce83","Type":"ContainerStarted","Data":"1d3494aa0bd7b26cbd1e49bd6a51c920184696d8c8d75847b0a5e7e124fbd9da"}
Jul 27 20:26:06.173716 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.173645    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-twglv" podUID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" containerName="authorino" containerID="cri-o://a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9" gracePeriod=30
Jul 27 20:26:06.174941 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.174917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" event={"ID":"da21c4ab-2fe0-4927-ac92-87f8817cae05","Type":"ContainerStarted","Data":"484fd50d0a8451dac6d8433ca3d1959610b66bc1425779584824de484c99363a"}
Jul 27 20:26:06.189192 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.189149    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-twglv" podStartSLOduration=0.807805751 podStartE2EDuration="1.189133996s" totalImagesPullingTime="381.328245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:26:06.188265874 +0000 UTC m=+499.038032706" watchObservedRunningTime="2026-07-27 20:26:06.189133996 +0000 UTC m=+499.038900827"
Jul 27 20:26:06.287133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.287099    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:26:06.289828 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:26:06.289761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac5825d_38b7_472d_a644_a6574044ccb8.slice/crio-0ad45bec883e1cf1a438885daabf8041b4864daf58c2ae702b54bdb9f08c0fa3 WatchSource:0}: Error finding container 0ad45bec883e1cf1a438885daabf8041b4864daf58c2ae702b54bdb9f08c0fa3: Status 404 returned error can't find the container with id 0ad45bec883e1cf1a438885daabf8041b4864daf58c2ae702b54bdb9f08c0fa3
Jul 27 20:26:06.410395 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.410373    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:06.470546 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.470513    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfv46\" (UniqueName: \"kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46\") pod \"303a6c33-839a-4d09-b7ee-75ad2a82ce83\" (UID: \"303a6c33-839a-4d09-b7ee-75ad2a82ce83\") "
Jul 27 20:26:06.472495 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.472468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46" (OuterVolumeSpecName: "kube-api-access-tfv46") pod "303a6c33-839a-4d09-b7ee-75ad2a82ce83" (UID: "303a6c33-839a-4d09-b7ee-75ad2a82ce83"). InnerVolumeSpecName "kube-api-access-tfv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:26:06.571349 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:06.571311    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfv46\" (UniqueName: \"kubernetes.io/projected/303a6c33-839a-4d09-b7ee-75ad2a82ce83-kube-api-access-tfv46\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:26:07.181032 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.181002    2560 generic.go:356] "Generic (PLEG): container finished" podID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" containerID="a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9" exitCode=2
Jul 27 20:26:07.181223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.181042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-twglv"
Jul 27 20:26:07.181223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.181082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-twglv" event={"ID":"303a6c33-839a-4d09-b7ee-75ad2a82ce83","Type":"ContainerDied","Data":"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"}
Jul 27 20:26:07.181223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.181121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-twglv" event={"ID":"303a6c33-839a-4d09-b7ee-75ad2a82ce83","Type":"ContainerDied","Data":"1d3494aa0bd7b26cbd1e49bd6a51c920184696d8c8d75847b0a5e7e124fbd9da"}
Jul 27 20:26:07.181223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.181143    2560 scope.go:117] "RemoveContainer" containerID="a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"
Jul 27 20:26:07.182789 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.182762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" event={"ID":"da21c4ab-2fe0-4927-ac92-87f8817cae05","Type":"ContainerStarted","Data":"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"}
Jul 27 20:26:07.182789 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.182771    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" podUID="da21c4ab-2fe0-4927-ac92-87f8817cae05" containerName="authorino" containerID="cri-o://d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42" gracePeriod=30
Jul 27 20:26:07.184417 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.184392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" event={"ID":"eac5825d-38b7-472d-a644-a6574044ccb8","Type":"ContainerStarted","Data":"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"}
Jul 27 20:26:07.184605 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.184432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" event={"ID":"eac5825d-38b7-472d-a644-a6574044ccb8","Type":"ContainerStarted","Data":"0ad45bec883e1cf1a438885daabf8041b4864daf58c2ae702b54bdb9f08c0fa3"}
Jul 27 20:26:07.190474 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.190456    2560 scope.go:117] "RemoveContainer" containerID="a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"
Jul 27 20:26:07.190722 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:26:07.190707    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9\": container with ID starting with a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9 not found: ID does not exist" containerID="a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"
Jul 27 20:26:07.190772 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.190729    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9"} err="failed to get container status \"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9\": rpc error: code = NotFound desc = could not find container \"a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9\": container with ID starting with a38a3a353c783901eedf702cc6bb8ca776526ecbacc7fcd233f1e619ff29bfe9 not found: ID does not exist"
Jul 27 20:26:07.220338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.220294    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" podStartSLOduration=1.890944328 podStartE2EDuration="2.220280662s" totalImagesPullingTime="329.336334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:26:07.219495714 +0000 UTC m=+500.069262546" watchObservedRunningTime="2026-07-27 20:26:07.220280662 +0000 UTC m=+500.070047513"
Jul 27 20:26:07.220945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.220922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" podStartSLOduration=1.819435006 podStartE2EDuration="2.220916911s" totalImagesPullingTime="401.481905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:26:07.199742429 +0000 UTC m=+500.049509259" watchObservedRunningTime="2026-07-27 20:26:07.220916911 +0000 UTC m=+500.070683742"
Jul 27 20:26:07.245026 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.244999    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:07.249777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.249751    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-twglv"]
Jul 27 20:26:07.408461 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.408440    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:07.478336 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.478307    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6c5v\" (UniqueName: \"kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v\") pod \"da21c4ab-2fe0-4927-ac92-87f8817cae05\" (UID: \"da21c4ab-2fe0-4927-ac92-87f8817cae05\") "
Jul 27 20:26:07.480288 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.480264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v" (OuterVolumeSpecName: "kube-api-access-n6c5v") pod "da21c4ab-2fe0-4927-ac92-87f8817cae05" (UID: "da21c4ab-2fe0-4927-ac92-87f8817cae05"). InnerVolumeSpecName "kube-api-access-n6c5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:26:07.579277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.579252    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6c5v\" (UniqueName: \"kubernetes.io/projected/da21c4ab-2fe0-4927-ac92-87f8817cae05-kube-api-access-n6c5v\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:26:07.726521 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:07.726448    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" path="/var/lib/kubelet/pods/303a6c33-839a-4d09-b7ee-75ad2a82ce83/volumes"
Jul 27 20:26:08.189858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.189826    2560 generic.go:356] "Generic (PLEG): container finished" podID="da21c4ab-2fe0-4927-ac92-87f8817cae05" containerID="d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42" exitCode=0
Jul 27 20:26:08.189858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.189862    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt"
Jul 27 20:26:08.190088 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.189920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" event={"ID":"da21c4ab-2fe0-4927-ac92-87f8817cae05","Type":"ContainerDied","Data":"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"}
Jul 27 20:26:08.190088 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.189961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6986bb9f4d-9l9mt" event={"ID":"da21c4ab-2fe0-4927-ac92-87f8817cae05","Type":"ContainerDied","Data":"484fd50d0a8451dac6d8433ca3d1959610b66bc1425779584824de484c99363a"}
Jul 27 20:26:08.190088 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.189985    2560 scope.go:117] "RemoveContainer" containerID="d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"
Jul 27 20:26:08.198259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.198245    2560 scope.go:117] "RemoveContainer" containerID="d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"
Jul 27 20:26:08.198528 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:26:08.198509    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42\": container with ID starting with d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42 not found: ID does not exist" containerID="d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"
Jul 27 20:26:08.198574 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.198536    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42"} err="failed to get container status \"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42\": rpc error: code = NotFound desc = could not find container \"d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42\": container with ID starting with d8a109a6e2d4284481b215495a7e9549da5725d98f48ea849801c84f43e4cb42 not found: ID does not exist"
Jul 27 20:26:08.205971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.205951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:08.210423 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:08.210404    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6986bb9f4d-9l9mt"]
Jul 27 20:26:09.731251 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:26:09.731214    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da21c4ab-2fe0-4927-ac92-87f8817cae05" path="/var/lib/kubelet/pods/da21c4ab-2fe0-4927-ac92-87f8817cae05/volumes"
Jul 27 20:27:17.809573 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809536    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"]
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809876    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da21c4ab-2fe0-4927-ac92-87f8817cae05" containerName="authorino"
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809887    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="da21c4ab-2fe0-4927-ac92-87f8817cae05" containerName="authorino"
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809907    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" containerName="authorino"
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809912    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" containerName="authorino"
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809964    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="303a6c33-839a-4d09-b7ee-75ad2a82ce83" containerName="authorino"
Jul 27 20:27:17.810089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.809973    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="da21c4ab-2fe0-4927-ac92-87f8817cae05" containerName="authorino"
Jul 27 20:27:17.812004 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.811986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.816031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.816006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 20:27:17.816153 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.816110    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 20:27:17.816153 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.816124    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 27 20:27:17.816153 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.816130    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fvph7\""
Jul 27 20:27:17.821185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.821165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"]
Jul 27 20:27:17.859660 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.859825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.859825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b58f60-b348-4bde-9ae7-b22417484de1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.859825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.859825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.859990 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.859848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdxk\" (UniqueName: \"kubernetes.io/projected/24b58f60-b348-4bde-9ae7-b22417484de1-kube-api-access-tfdxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961243 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdxk\" (UniqueName: \"kubernetes.io/projected/24b58f60-b348-4bde-9ae7-b22417484de1-kube-api-access-tfdxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961243 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961506 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961506 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b58f60-b348-4bde-9ae7-b22417484de1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961601 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961601 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961699 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961914 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.961985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.961912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.963576 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.963555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b58f60-b348-4bde-9ae7-b22417484de1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.963926 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.963907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b58f60-b348-4bde-9ae7-b22417484de1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:17.969237 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:17.969212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdxk\" (UniqueName: \"kubernetes.io/projected/24b58f60-b348-4bde-9ae7-b22417484de1-kube-api-access-tfdxk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-czr7n\" (UID: \"24b58f60-b348-4bde-9ae7-b22417484de1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:18.122567 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:18.122534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:18.247275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:18.247252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"]
Jul 27 20:27:18.249447 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:27:18.249413    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b58f60_b348_4bde_9ae7_b22417484de1.slice/crio-b46f060a3209deea0bbfcd6feaa5951e437ca31c7fd2f6a3362db6481a75c1de WatchSource:0}: Error finding container b46f060a3209deea0bbfcd6feaa5951e437ca31c7fd2f6a3362db6481a75c1de: Status 404 returned error can't find the container with id b46f060a3209deea0bbfcd6feaa5951e437ca31c7fd2f6a3362db6481a75c1de
Jul 27 20:27:18.510136 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:18.510057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n" event={"ID":"24b58f60-b348-4bde-9ae7-b22417484de1","Type":"ContainerStarted","Data":"b46f060a3209deea0bbfcd6feaa5951e437ca31c7fd2f6a3362db6481a75c1de"}
Jul 27 20:27:19.516965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:19.516933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n" event={"ID":"24b58f60-b348-4bde-9ae7-b22417484de1","Type":"ContainerStarted","Data":"068c45399705e1e762744a5bbb1d9c6cbefc792e54fa4d03db7933e92bec0365"}
Jul 27 20:27:19.539862 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:19.539785    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n" podStartSLOduration=1.431691874 podStartE2EDuration="2.539768676s" totalImagesPullingTime="1.108076802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:27:19.537847929 +0000 UTC m=+572.387614757" watchObservedRunningTime="2026-07-27 20:27:19.539768676 +0000 UTC m=+572.389535505"
Jul 27 20:27:28.122973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.122941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:28.123408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.123022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:28.135122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.135100    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:28.568998 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.568913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-czr7n"
Jul 27 20:27:28.992555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.992526    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"]
Jul 27 20:27:28.996089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.996067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:28.998724 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:28.998702    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 20:27:29.006019 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.005995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"]
Jul 27 20:27:29.058595 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.058733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57kn\" (UniqueName: \"kubernetes.io/projected/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-kube-api-access-g57kn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.058733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.058826 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.058879 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.058922 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.058907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160283 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57kn\" (UniqueName: \"kubernetes.io/projected/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-kube-api-access-g57kn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.160741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.161033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.161033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.161033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.160980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.162768 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.162741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.163126 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.163107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.168144 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.168118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57kn\" (UniqueName: \"kubernetes.io/projected/1c689fb6-96bd-47f0-afbb-bf062cb2b94a-kube-api-access-g57kn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m7zwx\" (UID: \"1c689fb6-96bd-47f0-afbb-bf062cb2b94a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.307031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.306949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:29.441369 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.441344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"]
Jul 27 20:27:29.443572 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:27:29.443547    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c689fb6_96bd_47f0_afbb_bf062cb2b94a.slice/crio-3a441a39bfe5bea34aa4339219bb76c2c9c9604ef6a5e124887fa60b014951f0 WatchSource:0}: Error finding container 3a441a39bfe5bea34aa4339219bb76c2c9c9604ef6a5e124887fa60b014951f0: Status 404 returned error can't find the container with id 3a441a39bfe5bea34aa4339219bb76c2c9c9604ef6a5e124887fa60b014951f0
Jul 27 20:27:29.563827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:29.563737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx" event={"ID":"1c689fb6-96bd-47f0-afbb-bf062cb2b94a","Type":"ContainerStarted","Data":"3a441a39bfe5bea34aa4339219bb76c2c9c9604ef6a5e124887fa60b014951f0"}
Jul 27 20:27:30.570611 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:30.570576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx" event={"ID":"1c689fb6-96bd-47f0-afbb-bf062cb2b94a","Type":"ContainerStarted","Data":"c62f069d3833c4c380d41bc9f0f90558e372d9cb342b4d1311cf7f136424fbd9"}
Jul 27 20:27:30.589532 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:30.589479    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx" podStartSLOduration=2.332882754 podStartE2EDuration="2.58946166s" totalImagesPullingTime="256.578906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:27:30.58856432 +0000 UTC m=+583.438331149" watchObservedRunningTime="2026-07-27 20:27:30.58946166 +0000 UTC m=+583.439228492"
Jul 27 20:27:39.307882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:39.307846    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:39.308265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:39.307915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:39.320322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:39.320292    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:27:39.621705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:27:39.621683    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m7zwx"
Jul 27 20:29:46.236894 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.236804    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:29:46.237378 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.236987    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" podUID="eac5825d-38b7-472d-a644-a6574044ccb8" containerName="authorino" containerID="cri-o://96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835" gracePeriod=30
Jul 27 20:29:46.486521 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.486497    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:29:46.554901 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.554829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42wqx\" (UniqueName: \"kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx\") pod \"eac5825d-38b7-472d-a644-a6574044ccb8\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") "
Jul 27 20:29:46.554901 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.554897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert\") pod \"eac5825d-38b7-472d-a644-a6574044ccb8\" (UID: \"eac5825d-38b7-472d-a644-a6574044ccb8\") "
Jul 27 20:29:46.556864 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.556839    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx" (OuterVolumeSpecName: "kube-api-access-42wqx") pod "eac5825d-38b7-472d-a644-a6574044ccb8" (UID: "eac5825d-38b7-472d-a644-a6574044ccb8"). InnerVolumeSpecName "kube-api-access-42wqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:29:46.564490 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.564466    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "eac5825d-38b7-472d-a644-a6574044ccb8" (UID: "eac5825d-38b7-472d-a644-a6574044ccb8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:29:46.655514 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.655488    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42wqx\" (UniqueName: \"kubernetes.io/projected/eac5825d-38b7-472d-a644-a6574044ccb8-kube-api-access-42wqx\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:29:46.655514 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:46.655512    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eac5825d-38b7-472d-a644-a6574044ccb8-tls-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:29:47.169759 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.169721    2560 generic.go:356] "Generic (PLEG): container finished" podID="eac5825d-38b7-472d-a644-a6574044ccb8" containerID="96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835" exitCode=0
Jul 27 20:29:47.169964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.169781    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d695cb8d8-vn5cz"
Jul 27 20:29:47.169964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.169819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" event={"ID":"eac5825d-38b7-472d-a644-a6574044ccb8","Type":"ContainerDied","Data":"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"}
Jul 27 20:29:47.169964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.169882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d695cb8d8-vn5cz" event={"ID":"eac5825d-38b7-472d-a644-a6574044ccb8","Type":"ContainerDied","Data":"0ad45bec883e1cf1a438885daabf8041b4864daf58c2ae702b54bdb9f08c0fa3"}
Jul 27 20:29:47.169964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.169901    2560 scope.go:117] "RemoveContainer" containerID="96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"
Jul 27 20:29:47.178596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.178576    2560 scope.go:117] "RemoveContainer" containerID="96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"
Jul 27 20:29:47.178880 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:29:47.178860    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835\": container with ID starting with 96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835 not found: ID does not exist" containerID="96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"
Jul 27 20:29:47.178942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.178889    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835"} err="failed to get container status \"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835\": rpc error: code = NotFound desc = could not find container \"96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835\": container with ID starting with 96401b924520a389266bf988b95a8a6e98fe800d8bdd6f4b03f9b3e38cc47835 not found: ID does not exist"
Jul 27 20:29:47.190309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.190288    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:29:47.198982 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.198963    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d695cb8d8-vn5cz"]
Jul 27 20:29:47.725644 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:29:47.725613    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac5825d-38b7-472d-a644-a6574044ccb8" path="/var/lib/kubelet/pods/eac5825d-38b7-472d-a644-a6574044ccb8/volumes"
Jul 27 20:30:00.144006 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.143968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"]
Jul 27 20:30:00.144467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.144327    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac5825d-38b7-472d-a644-a6574044ccb8" containerName="authorino"
Jul 27 20:30:00.144467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.144337    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac5825d-38b7-472d-a644-a6574044ccb8" containerName="authorino"
Jul 27 20:30:00.144467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.144390    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac5825d-38b7-472d-a644-a6574044ccb8" containerName="authorino"
Jul 27 20:30:00.148548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.148531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"
Jul 27 20:30:00.151109 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.151088    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:30:00.154681 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.154645    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"]
Jul 27 20:30:00.159198 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.159179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"
Jul 27 20:30:00.286765 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.286730    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"]
Jul 27 20:30:00.288912 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:30:00.288884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6877e1_462c_4eae_91e1_830527e3c5ad.slice/crio-51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171 WatchSource:0}: Error finding container 51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171: Status 404 returned error can't find the container with id 51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171
Jul 27 20:30:00.290762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:00.290746    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:30:01.231383 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:01.231343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4" event={"ID":"8b6877e1-462c-4eae-91e1-830527e3c5ad","Type":"ContainerStarted","Data":"51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171"}
Jul 27 20:30:04.249833 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:04.249775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4" event={"ID":"8b6877e1-462c-4eae-91e1-830527e3c5ad","Type":"ContainerStarted","Data":"e8065734bf1a7b32a58f901101bde3298d3674b0e4d41fcaaae5bf4c3da091b7"}
Jul 27 20:30:04.265145 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:04.265083    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4" podStartSLOduration=1.103520915 podStartE2EDuration="4.265067896s" totalImagesPullingTime="3.161546981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:04.264309862 +0000 UTC m=+737.114076693" watchObservedRunningTime="2026-07-27 20:30:04.265067896 +0000 UTC m=+737.114834726"
Jul 27 20:30:05.255272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:05.255234    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b6877e1-462c-4eae-91e1-830527e3c5ad" containerID="e8065734bf1a7b32a58f901101bde3298d3674b0e4d41fcaaae5bf4c3da091b7" exitCode=0
Jul 27 20:30:05.255676 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:05.255323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4" event={"ID":"8b6877e1-462c-4eae-91e1-830527e3c5ad","Type":"ContainerDied","Data":"e8065734bf1a7b32a58f901101bde3298d3674b0e4d41fcaaae5bf4c3da091b7"}
Jul 27 20:30:06.384758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:06.384734    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"
Jul 27 20:30:07.267259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:07.267232    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4"
Jul 27 20:30:07.267259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:07.267240    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-x2mt4" event={"ID":"8b6877e1-462c-4eae-91e1-830527e3c5ad","Type":"ContainerDied","Data":"51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171"}
Jul 27 20:30:07.267513 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:30:07.267289    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51235a23ceb88be53fafb6476c263a54bca92a0ff9fde157d1f58e75f4a84171"
Jul 27 20:31:34.333880 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.333845    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:34.334291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.334182    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6877e1-462c-4eae-91e1-830527e3c5ad" containerName="cleanup"
Jul 27 20:31:34.334291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.334192    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6877e1-462c-4eae-91e1-830527e3c5ad" containerName="cleanup"
Jul 27 20:31:34.334291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.334263    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6877e1-462c-4eae-91e1-830527e3c5ad" containerName="cleanup"
Jul 27 20:31:34.337383 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.337363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.341678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.341642    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 20:31:34.341798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.341648    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 20:31:34.341798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.341729    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svv94\""
Jul 27 20:31:34.348852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.348829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:34.423453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.423419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9wx\" (UniqueName: \"kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.423612 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.423521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.524554 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.524513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.524735 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.524566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9wx\" (UniqueName: \"kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.527128 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.527103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.532942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.532918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9wx\" (UniqueName: \"kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx\") pod \"maas-controller-6cbc7b97fc-9xs7q\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") " pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.650600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.650570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:34.782932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:34.782871    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:34.785228 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:31:34.785201    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dd5bbc_e67b_469e_a5b0_5a1f8e702c9d.slice/crio-6f592e3e216c9fdd15abfd5f8cafd17caded3837f206e8102d27a324d132e578 WatchSource:0}: Error finding container 6f592e3e216c9fdd15abfd5f8cafd17caded3837f206e8102d27a324d132e578: Status 404 returned error can't find the container with id 6f592e3e216c9fdd15abfd5f8cafd17caded3837f206e8102d27a324d132e578
Jul 27 20:31:35.659559 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:35.659523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" event={"ID":"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d","Type":"ContainerStarted","Data":"6f592e3e216c9fdd15abfd5f8cafd17caded3837f206e8102d27a324d132e578"}
Jul 27 20:31:37.671463 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:37.671388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" event={"ID":"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d","Type":"ContainerStarted","Data":"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"}
Jul 27 20:31:37.671932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:37.671519    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:37.693532 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:37.693490    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" podStartSLOduration=0.981240483 podStartE2EDuration="3.693476429s" totalImagesPullingTime="2.712235946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:31:37.691854566 +0000 UTC m=+830.541621397" watchObservedRunningTime="2026-07-27 20:31:37.693476429 +0000 UTC m=+830.543243259"
Jul 27 20:31:48.679168 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:48.679136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:53.403948 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.403910    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:53.404342 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.404112    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" podUID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" containerName="manager" containerID="cri-o://690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5" gracePeriod=30
Jul 27 20:31:53.648085 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.648063    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:53.692085 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.692004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert\") pod \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") "
Jul 27 20:31:53.692085 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.692077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr9wx\" (UniqueName: \"kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx\") pod \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\" (UID: \"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d\") "
Jul 27 20:31:53.694309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.694277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" (UID: "02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:31:53.694309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.694299    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx" (OuterVolumeSpecName: "kube-api-access-dr9wx") pod "02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" (UID: "02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d"). InnerVolumeSpecName "kube-api-access-dr9wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:31:53.745923 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.745890    2560 generic.go:356] "Generic (PLEG): container finished" podID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" containerID="690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5" exitCode=0
Jul 27 20:31:53.746023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.745942    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q"
Jul 27 20:31:53.746023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.745954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" event={"ID":"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d","Type":"ContainerDied","Data":"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"}
Jul 27 20:31:53.746023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.746003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-9xs7q" event={"ID":"02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d","Type":"ContainerDied","Data":"6f592e3e216c9fdd15abfd5f8cafd17caded3837f206e8102d27a324d132e578"}
Jul 27 20:31:53.746123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.746032    2560 scope.go:117] "RemoveContainer" containerID="690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"
Jul 27 20:31:53.754077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.754058    2560 scope.go:117] "RemoveContainer" containerID="690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"
Jul 27 20:31:53.754347 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:31:53.754324    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5\": container with ID starting with 690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5 not found: ID does not exist" containerID="690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"
Jul 27 20:31:53.754408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.754357    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5"} err="failed to get container status \"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5\": rpc error: code = NotFound desc = could not find container \"690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5\": container with ID starting with 690716f15c430e84098e5a22d5e574b521651947d456d70c14b4f3fb55634bd5 not found: ID does not exist"
Jul 27 20:31:53.765027 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.765005    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:53.767239 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.767220    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-9xs7q"]
Jul 27 20:31:53.792715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.792697    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-webhook-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:31:53.792715 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:53.792714    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9wx\" (UniqueName: \"kubernetes.io/projected/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d-kube-api-access-dr9wx\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:31:54.652341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.652307    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-8xs5p"]
Jul 27 20:31:54.652716 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.652659    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" containerName="manager"
Jul 27 20:31:54.652716 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.652669    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" containerName="manager"
Jul 27 20:31:54.652791 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.652725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" containerName="manager"
Jul 27 20:31:54.656944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.656928    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.660544 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.660524    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 20:31:54.660674 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.660547    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-svv94\""
Jul 27 20:31:54.660674 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.660525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 20:31:54.665108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.665084    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-8xs5p"]
Jul 27 20:31:54.698917 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.698890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-webhook-cert\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.698917 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.698920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552wd\" (UniqueName: \"kubernetes.io/projected/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-kube-api-access-552wd\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.800179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.800142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552wd\" (UniqueName: \"kubernetes.io/projected/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-kube-api-access-552wd\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.800395 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.800243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-webhook-cert\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.802789 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.802758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-webhook-cert\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.809039 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.809014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552wd\" (UniqueName: \"kubernetes.io/projected/9cacd101-c89a-4fd0-9749-6e06c4f5eed9-kube-api-access-552wd\") pod \"maas-controller-6cbc7b97fc-8xs5p\" (UID: \"9cacd101-c89a-4fd0-9749-6e06c4f5eed9\") " pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:54.969956 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:54.969867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:55.312939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:55.312914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cbc7b97fc-8xs5p"]
Jul 27 20:31:55.314397 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:31:55.314371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cacd101_c89a_4fd0_9749_6e06c4f5eed9.slice/crio-c87c718a62df153394edede8f2f8938aad8201d5b7e2b11182a413de675604ee WatchSource:0}: Error finding container c87c718a62df153394edede8f2f8938aad8201d5b7e2b11182a413de675604ee: Status 404 returned error can't find the container with id c87c718a62df153394edede8f2f8938aad8201d5b7e2b11182a413de675604ee
Jul 27 20:31:55.726328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:55.726289    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d" path="/var/lib/kubelet/pods/02dd5bbc-e67b-469e-a5b0-5a1f8e702c9d/volumes"
Jul 27 20:31:55.760138 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:55.760091    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p" event={"ID":"9cacd101-c89a-4fd0-9749-6e06c4f5eed9","Type":"ContainerStarted","Data":"c87c718a62df153394edede8f2f8938aad8201d5b7e2b11182a413de675604ee"}
Jul 27 20:31:56.767411 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:56.767375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p" event={"ID":"9cacd101-c89a-4fd0-9749-6e06c4f5eed9","Type":"ContainerStarted","Data":"aad42a54f5883c7f5b04f335c74d7853e5e67e9d1504c79199a89b2d03b792f1"}
Jul 27 20:31:56.767905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:56.767481    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:31:56.785483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:31:56.785436    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p" podStartSLOduration=2.260975898 podStartE2EDuration="2.78542115s" totalImagesPullingTime="524.445252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:31:56.783623754 +0000 UTC m=+849.633390583" watchObservedRunningTime="2026-07-27 20:31:56.78542115 +0000 UTC m=+849.635187959"
Jul 27 20:32:07.774487 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:32:07.774457    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cbc7b97fc-8xs5p"
Jul 27 20:36:31.175720 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:31.175685    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:36:31.176216 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:31.175904    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" podUID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" containerName="manager" containerID="cri-o://71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38" gracePeriod=10
Jul 27 20:36:33.719077 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.719055    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:36:33.833089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.833019    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume\") pod \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") "
Jul 27 20:36:33.833089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.833064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-676jz\" (UniqueName: \"kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz\") pod \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\" (UID: \"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a\") "
Jul 27 20:36:33.833382 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.833345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" (UID: "1bdc25fa-2db2-4573-9e12-cf91ae3fab9a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:36:33.835052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.835022    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz" (OuterVolumeSpecName: "kube-api-access-676jz") pod "1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" (UID: "1bdc25fa-2db2-4573-9e12-cf91ae3fab9a"). InnerVolumeSpecName "kube-api-access-676jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:33.934523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.934493    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-extensions-socket-volume\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:36:33.934523 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:33.934519    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676jz\" (UniqueName: \"kubernetes.io/projected/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a-kube-api-access-676jz\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:36:34.036349 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.036315    2560 generic.go:356] "Generic (PLEG): container finished" podID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" containerID="71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38" exitCode=0
Jul 27 20:36:34.036507 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.036357    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"
Jul 27 20:36:34.036507 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.036392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" event={"ID":"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a","Type":"ContainerDied","Data":"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"}
Jul 27 20:36:34.036507 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.036445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l" event={"ID":"1bdc25fa-2db2-4573-9e12-cf91ae3fab9a","Type":"ContainerDied","Data":"d5bc91ad9f5ee410dc4ff7b38011e9920624c82227a86e9a19eb628f3aa1f597"}
Jul 27 20:36:34.036507 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.036468    2560 scope.go:117] "RemoveContainer" containerID="71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"
Jul 27 20:36:34.045545 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.045525    2560 scope.go:117] "RemoveContainer" containerID="71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"
Jul 27 20:36:34.045787 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:36:34.045771    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38\": container with ID starting with 71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38 not found: ID does not exist" containerID="71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"
Jul 27 20:36:34.045858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.045793    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38"} err="failed to get container status \"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38\": rpc error: code = NotFound desc = could not find container \"71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38\": container with ID starting with 71f532a2c5afe96f437db387b26cb1a35bb4091319dd394fa681928e74bf7b38 not found: ID does not exist"
Jul 27 20:36:34.062687 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.062664    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:36:34.070123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:34.070102    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z9k8l"]
Jul 27 20:36:35.726058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:36:35.726028    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" path="/var/lib/kubelet/pods/1bdc25fa-2db2-4573-9e12-cf91ae3fab9a/volumes"
Jul 27 20:37:32.358756 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.358723    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"]
Jul 27 20:37:32.359234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.359082    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" containerName="manager"
Jul 27 20:37:32.359234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.359094    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" containerName="manager"
Jul 27 20:37:32.359234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.359158    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdc25fa-2db2-4573-9e12-cf91ae3fab9a" containerName="manager"
Jul 27 20:37:32.362146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.362130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.366261 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.365991    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:37:32.366261 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.366165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:37:32.366469 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.366281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-55ddz\""
Jul 27 20:37:32.378741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.378715    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"]
Jul 27 20:37:32.526416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.526379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.526578 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.526434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bptwp\" (UniqueName: \"kubernetes.io/projected/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-kube-api-access-bptwp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.627259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.627230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.627417 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.627274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bptwp\" (UniqueName: \"kubernetes.io/projected/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-kube-api-access-bptwp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.627631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.627611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.641004 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.640984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bptwp\" (UniqueName: \"kubernetes.io/projected/45e0ccdf-6f35-40cb-a662-2fa1c20cef28-kube-api-access-bptwp\") pod \"kuadrant-operator-controller-manager-86878c769f-qrn8r\" (UID: \"45e0ccdf-6f35-40cb-a662-2fa1c20cef28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.676936 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.676912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:32.816604 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:32.816579    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"]
Jul 27 20:37:32.819267 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:37:32.819240    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e0ccdf_6f35_40cb_a662_2fa1c20cef28.slice/crio-400da2d60656e7817fe7cf45305453bd3b170c786c119b3d3f9a690e839cc0d8 WatchSource:0}: Error finding container 400da2d60656e7817fe7cf45305453bd3b170c786c119b3d3f9a690e839cc0d8: Status 404 returned error can't find the container with id 400da2d60656e7817fe7cf45305453bd3b170c786c119b3d3f9a690e839cc0d8
Jul 27 20:37:33.291719 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:33.291629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r" event={"ID":"45e0ccdf-6f35-40cb-a662-2fa1c20cef28","Type":"ContainerStarted","Data":"0cae81c46b8221551ac7438d004f004c0167c01488e750304038a10188f11947"}
Jul 27 20:37:33.291719 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:33.291687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r" event={"ID":"45e0ccdf-6f35-40cb-a662-2fa1c20cef28","Type":"ContainerStarted","Data":"400da2d60656e7817fe7cf45305453bd3b170c786c119b3d3f9a690e839cc0d8"}
Jul 27 20:37:33.291719 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:33.291715    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:37:33.314873 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:33.314796    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r" podStartSLOduration=1.314778456 podStartE2EDuration="1.314778456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:33.313025516 +0000 UTC m=+1186.162792363" watchObservedRunningTime="2026-07-27 20:37:33.314778456 +0000 UTC m=+1186.164545287"
Jul 27 20:37:44.299186 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:37:44.299154    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qrn8r"
Jul 27 20:40:21.441571 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.441497    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:21.444903 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.444889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.448046 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.448024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7c30250e-kserve-self-signed-certs\""
Jul 27 20:40:21.476680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.476649    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:21.529089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.529265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.529265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.529265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.529265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.529438 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.529345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6ll\" (UniqueName: \"kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6ll\" (UniqueName: \"kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630481 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630481 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630481 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630641 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630641 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630881 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.630953 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.631003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.630967    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.632739 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.632714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.633038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.633023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.640381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.640348    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6ll\" (UniqueName: \"kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll\") pod \"e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") " pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.753643 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.753557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:21.900238 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:21.900211    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:21.904980 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:40:21.904954    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod465dc8ae_9aae_4c0f_9835_9410c75f94bd.slice/crio-cc3f1dc0159569af13ccc5be3d01d55e5c17373870fec1685b0269a4091ab68c WatchSource:0}: Error finding container cc3f1dc0159569af13ccc5be3d01d55e5c17373870fec1685b0269a4091ab68c: Status 404 returned error can't find the container with id cc3f1dc0159569af13ccc5be3d01d55e5c17373870fec1685b0269a4091ab68c
Jul 27 20:40:22.084328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:22.084235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" event={"ID":"465dc8ae-9aae-4c0f-9835-9410c75f94bd","Type":"ContainerStarted","Data":"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"}
Jul 27 20:40:22.084328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:22.084278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" event={"ID":"465dc8ae-9aae-4c0f-9835-9410c75f94bd","Type":"ContainerStarted","Data":"cc3f1dc0159569af13ccc5be3d01d55e5c17373870fec1685b0269a4091ab68c"}
Jul 27 20:40:22.128556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:22.128504    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" podStartSLOduration=1.128487942 podStartE2EDuration="1.128487942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:40:22.125372496 +0000 UTC m=+1354.975139327" watchObservedRunningTime="2026-07-27 20:40:22.128487942 +0000 UTC m=+1354.978254796"
Jul 27 20:40:24.008791 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.008755    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:24.012853 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.012826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.017707 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.017685    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7c30250e-kserve-self-signed-certs\""
Jul 27 20:40:24.025973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.025950    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:24.153002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.152967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.153002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.153003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk5b\" (UniqueName: \"kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.153248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.153121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.153248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.153185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.153248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.153230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.153393 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.153308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254327 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254327 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254562 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254562 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254562 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254562 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk5b\" (UniqueName: \"kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254766 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254766 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.254960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.254934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.256616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.256599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.256777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.256762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.267018 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.266949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk5b\" (UniqueName: \"kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b\") pod \"e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") " pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.324844 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.324793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:24.485322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:24.485283    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:24.490742 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:40:24.490715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod953339be_ba44_4c1f_b404_d22bddf8f2a3.slice/crio-4980889ed8f12da19092424d7822a6893a1a5136bc9569faf1b4f5b5cbf40835 WatchSource:0}: Error finding container 4980889ed8f12da19092424d7822a6893a1a5136bc9569faf1b4f5b5cbf40835: Status 404 returned error can't find the container with id 4980889ed8f12da19092424d7822a6893a1a5136bc9569faf1b4f5b5cbf40835
Jul 27 20:40:25.100402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:25.100361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" event={"ID":"953339be-ba44-4c1f-b404-d22bddf8f2a3","Type":"ContainerStarted","Data":"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"}
Jul 27 20:40:25.100878 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:25.100415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" event={"ID":"953339be-ba44-4c1f-b404-d22bddf8f2a3","Type":"ContainerStarted","Data":"4980889ed8f12da19092424d7822a6893a1a5136bc9569faf1b4f5b5cbf40835"}
Jul 27 20:40:25.122824 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:25.122751    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" podStartSLOduration=2.122734354 podStartE2EDuration="2.122734354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:40:25.121544354 +0000 UTC m=+1357.971311185" watchObservedRunningTime="2026-07-27 20:40:25.122734354 +0000 UTC m=+1357.972501183"
Jul 27 20:40:31.754324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:31.754249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:31.754324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:31.754289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:31.766412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:31.766382    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:32.141894 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:32.141865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:34.324997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:34.324967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:34.324997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:34.325013    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:34.337183 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:34.337162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:35.155112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:35.155082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:40.965463 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:40.965424    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:42.653423 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:42.653387    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:55.984284 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:55.984220    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" podUID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" containerName="main" containerID="cri-o://f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413" gracePeriod=45
Jul 27 20:40:56.231869 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.231844    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:56.238734 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.238659    2560 generic.go:356] "Generic (PLEG): container finished" podID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" containerID="f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413" exitCode=0
Jul 27 20:40:56.238734 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.238695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"
Jul 27 20:40:56.238734 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.238716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" event={"ID":"465dc8ae-9aae-4c0f-9835-9410c75f94bd","Type":"ContainerDied","Data":"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"}
Jul 27 20:40:56.238957 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.238744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s" event={"ID":"465dc8ae-9aae-4c0f-9835-9410c75f94bd","Type":"ContainerDied","Data":"cc3f1dc0159569af13ccc5be3d01d55e5c17373870fec1685b0269a4091ab68c"}
Jul 27 20:40:56.238957 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.238761    2560 scope.go:117] "RemoveContainer" containerID="f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"
Jul 27 20:40:56.246929 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.246905    2560 scope.go:117] "RemoveContainer" containerID="f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"
Jul 27 20:40:56.247202 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:40:56.247185    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413\": container with ID starting with f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413 not found: ID does not exist" containerID="f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"
Jul 27 20:40:56.247284 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.247211    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413"} err="failed to get container status \"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413\": rpc error: code = NotFound desc = could not find container \"f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413\": container with ID starting with f213f0e3cd8fcc0cf0dece13e566e6c3d01a890aa87f48f8817901b4b910d413 not found: ID does not exist"
Jul 27 20:40:56.318728 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.318728 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.318992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318756    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.318992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.318992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318824    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.318992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.318844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq6ll\" (UniqueName: \"kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll\") pod \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\" (UID: \"465dc8ae-9aae-4c0f-9835-9410c75f94bd\") "
Jul 27 20:40:56.319187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.319066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:56.319187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.319070    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home" (OuterVolumeSpecName: "home") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:56.319187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.319108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:56.321075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.321043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:40:56.321181 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.321084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm" (OuterVolumeSpecName: "dshm") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:56.321181 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.321128    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll" (OuterVolumeSpecName: "kube-api-access-zq6ll") pod "465dc8ae-9aae-4c0f-9835-9410c75f94bd" (UID: "465dc8ae-9aae-4c0f-9835-9410c75f94bd"). InnerVolumeSpecName "kube-api-access-zq6ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:40:56.419394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419361    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.419394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419387    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6ll\" (UniqueName: \"kubernetes.io/projected/465dc8ae-9aae-4c0f-9835-9410c75f94bd-kube-api-access-zq6ll\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.419394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419396    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.419394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419404    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.419653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419412    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465dc8ae-9aae-4c0f-9835-9410c75f94bd-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.419653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.419420    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465dc8ae-9aae-4c0f-9835-9410c75f94bd-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:56.562467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.562434    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:56.579834 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:56.575953    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7c30250e-kserve-674db7c9cf-sv76s"]
Jul 27 20:40:57.679007 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.678970    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" podUID="953339be-ba44-4c1f-b404-d22bddf8f2a3" containerName="main" containerID="cri-o://87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7" gracePeriod=45
Jul 27 20:40:57.725824 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.725775    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" path="/var/lib/kubelet/pods/465dc8ae-9aae-4c0f-9835-9410c75f94bd/volumes"
Jul 27 20:40:57.925150 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.925125    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:57.930134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930072    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930293 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930129    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930293 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930293 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930216    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtk5b\" (UniqueName: \"kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930494 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930377    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930494 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:57.930494 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home\") pod \"953339be-ba44-4c1f-b404-d22bddf8f2a3\" (UID: \"953339be-ba44-4c1f-b404-d22bddf8f2a3\") "
Jul 27 20:40:57.930739 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930625    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:57.930739 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home" (OuterVolumeSpecName: "home") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:57.931033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930804    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:57.931033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.930839    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:57.932298 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.932278    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm" (OuterVolumeSpecName: "dshm") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:40:57.932386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.932315    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:40:57.932386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:57.932312    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b" (OuterVolumeSpecName: "kube-api-access-rtk5b") pod "953339be-ba44-4c1f-b404-d22bddf8f2a3" (UID: "953339be-ba44-4c1f-b404-d22bddf8f2a3"). InnerVolumeSpecName "kube-api-access-rtk5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:40:58.031289 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.031263    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:58.031289 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.031286    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/953339be-ba44-4c1f-b404-d22bddf8f2a3-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:58.031289 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.031295    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/953339be-ba44-4c1f-b404-d22bddf8f2a3-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:58.031517 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.031304    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtk5b\" (UniqueName: \"kubernetes.io/projected/953339be-ba44-4c1f-b404-d22bddf8f2a3-kube-api-access-rtk5b\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:40:58.250133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.250044    2560 generic.go:356] "Generic (PLEG): container finished" podID="953339be-ba44-4c1f-b404-d22bddf8f2a3" containerID="87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7" exitCode=0
Jul 27 20:40:58.250133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.250083    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" event={"ID":"953339be-ba44-4c1f-b404-d22bddf8f2a3","Type":"ContainerDied","Data":"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"}
Jul 27 20:40:58.250133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.250084    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"
Jul 27 20:40:58.250133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.250112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84" event={"ID":"953339be-ba44-4c1f-b404-d22bddf8f2a3","Type":"ContainerDied","Data":"4980889ed8f12da19092424d7822a6893a1a5136bc9569faf1b4f5b5cbf40835"}
Jul 27 20:40:58.250133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.250128    2560 scope.go:117] "RemoveContainer" containerID="87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"
Jul 27 20:40:58.258677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.258660    2560 scope.go:117] "RemoveContainer" containerID="87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"
Jul 27 20:40:58.258952 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:40:58.258935    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7\": container with ID starting with 87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7 not found: ID does not exist" containerID="87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"
Jul 27 20:40:58.259003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.258961    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7"} err="failed to get container status \"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7\": rpc error: code = NotFound desc = could not find container \"87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7\": container with ID starting with 87778c37bcde4de23cc5c02d64d1e387458791af396e2d301379ce1e6c50bed7 not found: ID does not exist"
Jul 27 20:40:58.272730 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.272706    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:58.281673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:58.281648    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7c30250e-kserve-d7664c76c-wpd84"]
Jul 27 20:40:59.726590 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:40:59.726559    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953339be-ba44-4c1f-b404-d22bddf8f2a3" path="/var/lib/kubelet/pods/953339be-ba44-4c1f-b404-d22bddf8f2a3/volumes"
Jul 27 20:44:23.399180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399145    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399492    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953339be-ba44-4c1f-b404-d22bddf8f2a3" containerName="main"
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399502    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="953339be-ba44-4c1f-b404-d22bddf8f2a3" containerName="main"
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399522    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" containerName="main"
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399527    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" containerName="main"
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399576    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="465dc8ae-9aae-4c0f-9835-9410c75f94bd" containerName="main"
Jul 27 20:44:23.399646 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.399585    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="953339be-ba44-4c1f-b404-d22bddf8f2a3" containerName="main"
Jul 27 20:44:23.402724 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.402709    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.405300 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.405279    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-584365a2-openshift-default-dockercfg-lv5pt\""
Jul 27 20:44:23.405428 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.405302    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:44:23.420835 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.420795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:44:23.510045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xzh\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.510548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.510359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.611755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.611719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.611755 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.611753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.611786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.611892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.611979    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xzh\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612352 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612539 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612612 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.612713 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.612692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.614094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.614072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.614314 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.614297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.622232 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.622207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.622232 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.622231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xzh\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh\") pod \"e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") " pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.712984 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.712899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:23.848883 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.848856    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:44:23.851484 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:44:23.851458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bfc49d_8788_401b_900d_c3aa51d51822.slice/crio-6137f3cadb1d4e5ccd843c7f00be1bb9de5f1e17931f11e3db2d87273e6f26a0 WatchSource:0}: Error finding container 6137f3cadb1d4e5ccd843c7f00be1bb9de5f1e17931f11e3db2d87273e6f26a0: Status 404 returned error can't find the container with id 6137f3cadb1d4e5ccd843c7f00be1bb9de5f1e17931f11e3db2d87273e6f26a0
Jul 27 20:44:23.853375 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:23.853359    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:44:24.195765 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:24.195733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerStarted","Data":"6137f3cadb1d4e5ccd843c7f00be1bb9de5f1e17931f11e3db2d87273e6f26a0"}
Jul 27 20:44:26.579871 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:26.579822    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:26.580187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:26.579898    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:26.580187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:26.579929    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:27.212640 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:27.212605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerStarted","Data":"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"}
Jul 27 20:44:27.233585 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:27.233509    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podStartSLOduration=1.507471526 podStartE2EDuration="4.233494311s" totalImagesPullingTime="2.726022785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:27.232216776 +0000 UTC m=+1600.081983609" watchObservedRunningTime="2026-07-27 20:44:27.233494311 +0000 UTC m=+1600.083261141"
Jul 27 20:44:27.713438 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:27.713391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:27.717996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:27.717975    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:28.216772 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.216738    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:28.217709 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.217691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:28.831251 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.831216    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:44:28.834563 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.834545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.837213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.837184    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-584365a2-serving-cert\""
Jul 27 20:44:28.837339 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.837221    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:44:28.838307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.838287    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:44:28.838405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.838323    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:44:28.843946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.843924    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:44:28.859202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.859181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.859306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.859269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbbn\" (UniqueName: \"kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.859353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.859309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.960470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.960442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbbn\" (UniqueName: \"kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.960632 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.960492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.960632 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.960526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.960739 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:44:28.960639    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-serving-cert: secret "maas-api-e2e-ait-584365a2-serving-cert" not found
Jul 27 20:44:28.960739 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:44:28.960723    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls podName:bf01b1a5-1ce0-422c-9a92-e28b91dc5617 nodeName:}" failed. No retries permitted until 2026-07-27 20:44:29.460694394 +0000 UTC m=+1602.310461207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls") pod "maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" (UID: "bf01b1a5-1ce0-422c-9a92-e28b91dc5617") : secret "maas-api-e2e-ait-584365a2-serving-cert" not found
Jul 27 20:44:28.961235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.961218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:28.969070 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:28.969045    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbbn\" (UniqueName: \"kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:29.466825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.466790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:29.469137 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.469114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") pod \"maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:29.508685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.508654    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:44:29.512402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.512382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.515256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.515238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-584365a2\""
Jul 27 20:44:29.515353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.515240    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-584365a2-dockercfg-d6mv6\""
Jul 27 20:44:29.523568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.523546    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:44:29.568137 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.568096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q7d\" (UniqueName: \"kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.568316 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.568227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.669286 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.669249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q7d\" (UniqueName: \"kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.669493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.669336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.670127 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.670106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.681083 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.678423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q7d\" (UniqueName: \"kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d\") pod \"payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.746305 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.746219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:29.824414 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.824377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:29.886389 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.886348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:44:29.888013 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:44:29.887973    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf01b1a5_1ce0_422c_9a92_e28b91dc5617.slice/crio-e51bf2b0cbb363afff6a81fbff0218d800438cf65015eff5e0547b369a930c64 WatchSource:0}: Error finding container e51bf2b0cbb363afff6a81fbff0218d800438cf65015eff5e0547b369a930c64: Status 404 returned error can't find the container with id e51bf2b0cbb363afff6a81fbff0218d800438cf65015eff5e0547b369a930c64
Jul 27 20:44:29.968532 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:29.968500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:44:29.971537 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:44:29.971508    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e749a24_7271_4e99_b49f_eafbc0afbfce.slice/crio-7da50ea5b0b7365166c487b6d4d7068b376e9d6ab996706617b2f1a8fd2f4e4d WatchSource:0}: Error finding container 7da50ea5b0b7365166c487b6d4d7068b376e9d6ab996706617b2f1a8fd2f4e4d: Status 404 returned error can't find the container with id 7da50ea5b0b7365166c487b6d4d7068b376e9d6ab996706617b2f1a8fd2f4e4d
Jul 27 20:44:30.227148 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:30.227106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" event={"ID":"bf01b1a5-1ce0-422c-9a92-e28b91dc5617","Type":"ContainerStarted","Data":"e51bf2b0cbb363afff6a81fbff0218d800438cf65015eff5e0547b369a930c64"}
Jul 27 20:44:30.228268 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:30.228244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" event={"ID":"7e749a24-7271-4e99-b49f-eafbc0afbfce","Type":"ContainerStarted","Data":"7da50ea5b0b7365166c487b6d4d7068b376e9d6ab996706617b2f1a8fd2f4e4d"}
Jul 27 20:44:32.240764 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.240727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" event={"ID":"7e749a24-7271-4e99-b49f-eafbc0afbfce","Type":"ContainerStarted","Data":"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"}
Jul 27 20:44:32.241252 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.240802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:32.242198 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.242173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" event={"ID":"bf01b1a5-1ce0-422c-9a92-e28b91dc5617","Type":"ContainerStarted","Data":"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"}
Jul 27 20:44:32.242346 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.242322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:32.255834 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.255760    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" podStartSLOduration=1.373562644 podStartE2EDuration="3.255739006s" totalImagesPullingTime="1.882176362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:32.254921672 +0000 UTC m=+1605.104688504" watchObservedRunningTime="2026-07-27 20:44:32.255739006 +0000 UTC m=+1605.105505836"
Jul 27 20:44:32.270878 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:32.270820    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" podStartSLOduration=2.311344799 podStartE2EDuration="4.270789486s" totalImagesPullingTime="1.959444687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:32.269516527 +0000 UTC m=+1605.119283359" watchObservedRunningTime="2026-07-27 20:44:32.270789486 +0000 UTC m=+1605.120556317"
Jul 27 20:44:38.253019 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:38.252991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:44:41.290714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.290682    2560 generic.go:356] "Generic (PLEG): container finished" podID="57bfc49d-8788-401b-900d-c3aa51d51822" containerID="c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38" exitCode=0
Jul 27 20:44:41.291163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.290740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerDied","Data":"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"}
Jul 27 20:44:41.291163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.291137    2560 scope.go:117] "RemoveContainer" containerID="c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"
Jul 27 20:44:41.293174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.293144    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:41.293248 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.293214    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:41.293290 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:41.293247    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:44:42.296859 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:42.296827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerStarted","Data":"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"}
Jul 27 20:44:42.714029 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:42.713989    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:42.715051 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:42.715024    2560 patch_prober.go:28] interesting pod/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 27 20:44:42.715175 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:42.715078    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 27 20:44:43.217740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:43.217695    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:43.249530 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:43.249504    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:44:43.714172 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:43.714131    2560 patch_prober.go:28] interesting pod/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 27 20:44:43.714577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:43.714201    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 27 20:44:44.714030 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:44.713993    2560 patch_prober.go:28] interesting pod/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 27 20:44:44.714214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:44.714072    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 27 20:44:45.713407 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:45.713364    2560 patch_prober.go:28] interesting pod/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 27 20:44:45.713600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:45.713447    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 27 20:44:46.717142 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:46.717110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:46.718052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:46.718035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:44:54.450827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.450765    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"]
Jul 27 20:44:54.455590 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.455565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:54.459009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.458983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:44:54.461354 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.461331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"]
Jul 27 20:44:54.605923 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.605887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-584365a2-2x6kj\" (UID: \"97f9c1ab-8c31-4d44-9b9e-46978843203f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:54.707236 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.707138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-584365a2-2x6kj\" (UID: \"97f9c1ab-8c31-4d44-9b9e-46978843203f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:54.707783 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.707759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-584365a2-2x6kj\" (UID: \"97f9c1ab-8c31-4d44-9b9e-46978843203f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:54.768508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.768476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:54.901843 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:54.901647    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"]
Jul 27 20:44:54.904706 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:44:54.904678    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f9c1ab_8c31_4d44_9b9e_46978843203f.slice/crio-18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63 WatchSource:0}: Error finding container 18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63: Status 404 returned error can't find the container with id 18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63
Jul 27 20:44:55.367631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:55.367596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj" event={"ID":"97f9c1ab-8c31-4d44-9b9e-46978843203f","Type":"ContainerStarted","Data":"405968615ae077b1ff5b622d63c1fb6a53d17d31fa0de4e8df37abd351071c98"}
Jul 27 20:44:55.367791 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:55.367661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj" event={"ID":"97f9c1ab-8c31-4d44-9b9e-46978843203f","Type":"ContainerStarted","Data":"18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63"}
Jul 27 20:44:55.383234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:55.383172    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj" podStartSLOduration=1.383158378 podStartE2EDuration="1.383158378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:44:55.38190017 +0000 UTC m=+1628.231666997" watchObservedRunningTime="2026-07-27 20:44:55.383158378 +0000 UTC m=+1628.232925216"
Jul 27 20:44:56.374229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:56.374176    2560 generic.go:356] "Generic (PLEG): container finished" podID="97f9c1ab-8c31-4d44-9b9e-46978843203f" containerID="405968615ae077b1ff5b622d63c1fb6a53d17d31fa0de4e8df37abd351071c98" exitCode=0
Jul 27 20:44:56.374640 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:56.374253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj" event={"ID":"97f9c1ab-8c31-4d44-9b9e-46978843203f","Type":"ContainerDied","Data":"405968615ae077b1ff5b622d63c1fb6a53d17d31fa0de4e8df37abd351071c98"}
Jul 27 20:44:57.507333 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:57.507309    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:57.631747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:57.631666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca\") pod \"97f9c1ab-8c31-4d44-9b9e-46978843203f\" (UID: \"97f9c1ab-8c31-4d44-9b9e-46978843203f\") "
Jul 27 20:44:57.632106 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:57.632084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97f9c1ab-8c31-4d44-9b9e-46978843203f" (UID: "97f9c1ab-8c31-4d44-9b9e-46978843203f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:44:57.733277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:57.733252    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f9c1ab-8c31-4d44-9b9e-46978843203f-maas-api-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:44:58.390706 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:58.390666    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj" event={"ID":"97f9c1ab-8c31-4d44-9b9e-46978843203f","Type":"ContainerDied","Data":"18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63"}
Jul 27 20:44:58.390706 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:58.390685    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"
Jul 27 20:44:58.390950 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:44:58.390747    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f546cbb5858eb1eb5721aac61ca3ca8fda7b9bd256d805f0e7875f7642df63"
Jul 27 20:45:00.134572 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.134537    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"]
Jul 27 20:45:00.135048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.134945    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f9c1ab-8c31-4d44-9b9e-46978843203f" containerName="revoke-keys"
Jul 27 20:45:00.135048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.134958    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f9c1ab-8c31-4d44-9b9e-46978843203f" containerName="revoke-keys"
Jul 27 20:45:00.135048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.135026    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f9c1ab-8c31-4d44-9b9e-46978843203f" containerName="revoke-keys"
Jul 27 20:45:00.137915 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.137899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"
Jul 27 20:45:00.140416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.140394    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:45:00.147202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.147175    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"]
Jul 27 20:45:00.149750 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.149729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"
Jul 27 20:45:00.278174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.278030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"]
Jul 27 20:45:00.280509 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:45:00.280474    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbee42f7_106c_48a8_9b59_a00f24921b9b.slice/crio-1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44 WatchSource:0}: Error finding container 1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44: Status 404 returned error can't find the container with id 1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44
Jul 27 20:45:00.403137 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:00.403059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c" event={"ID":"bbee42f7-106c-48a8-9b59-a00f24921b9b","Type":"ContainerStarted","Data":"1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44"}
Jul 27 20:45:01.409230 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:01.409193    2560 generic.go:356] "Generic (PLEG): container finished" podID="bbee42f7-106c-48a8-9b59-a00f24921b9b" containerID="635b13f11e0f0504f460f94d624ebc158c96f95d3602e4a0195a73eda98cab96" exitCode=0
Jul 27 20:45:01.409650 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:01.409277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c" event={"ID":"bbee42f7-106c-48a8-9b59-a00f24921b9b","Type":"ContainerDied","Data":"635b13f11e0f0504f460f94d624ebc158c96f95d3602e4a0195a73eda98cab96"}
Jul 27 20:45:02.544997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:02.544971    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"
Jul 27 20:45:03.420245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:03.420214    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c"
Jul 27 20:45:03.420412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:03.420264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-mgm5c" event={"ID":"bbee42f7-106c-48a8-9b59-a00f24921b9b","Type":"ContainerDied","Data":"1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44"}
Jul 27 20:45:03.420412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:03.420308    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1854a120c239b2c67a61b5c4c5bfa0e8424c6b0c869af56f82b1a8e937432c44"
Jul 27 20:45:04.936882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:04.936847    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:45:04.937324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:04.937085    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" podUID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" containerName="maas-api" containerID="cri-o://1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb" gracePeriod=30
Jul 27 20:45:04.995107 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:04.995070    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:45:04.995433 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:04.995376    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" podUID="7e749a24-7271-4e99-b49f-eafbc0afbfce" containerName="payload-pre-processing" containerID="cri-o://6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766" gracePeriod=30
Jul 27 20:45:05.263285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.263258    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:45:05.282116 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.282089    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:45:05.402275 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca\") pod \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") "
Jul 27 20:45:05.402439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402304    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trbbn\" (UniqueName: \"kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn\") pod \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") "
Jul 27 20:45:05.402439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") pod \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\" (UID: \"bf01b1a5-1ce0-422c-9a92-e28b91dc5617\") "
Jul 27 20:45:05.402439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume\") pod \"7e749a24-7271-4e99-b49f-eafbc0afbfce\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") "
Jul 27 20:45:05.402439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402436    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2q7d\" (UniqueName: \"kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d\") pod \"7e749a24-7271-4e99-b49f-eafbc0afbfce\" (UID: \"7e749a24-7271-4e99-b49f-eafbc0afbfce\") "
Jul 27 20:45:05.402760 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf01b1a5-1ce0-422c-9a92-e28b91dc5617" (UID: "bf01b1a5-1ce0-422c-9a92-e28b91dc5617"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:05.402931 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.402869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e749a24-7271-4e99-b49f-eafbc0afbfce" (UID: "7e749a24-7271-4e99-b49f-eafbc0afbfce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:05.404595 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.404562    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf01b1a5-1ce0-422c-9a92-e28b91dc5617" (UID: "bf01b1a5-1ce0-422c-9a92-e28b91dc5617"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:45:05.404694 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.404616    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d" (OuterVolumeSpecName: "kube-api-access-r2q7d") pod "7e749a24-7271-4e99-b49f-eafbc0afbfce" (UID: "7e749a24-7271-4e99-b49f-eafbc0afbfce"). InnerVolumeSpecName "kube-api-access-r2q7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:05.404694 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.404640    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn" (OuterVolumeSpecName: "kube-api-access-trbbn") pod "bf01b1a5-1ce0-422c-9a92-e28b91dc5617" (UID: "bf01b1a5-1ce0-422c-9a92-e28b91dc5617"). InnerVolumeSpecName "kube-api-access-trbbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:05.431053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.431025    2560 generic.go:356] "Generic (PLEG): container finished" podID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" containerID="1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb" exitCode=0
Jul 27 20:45:05.431177 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.431075    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"
Jul 27 20:45:05.431177 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.431113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" event={"ID":"bf01b1a5-1ce0-422c-9a92-e28b91dc5617","Type":"ContainerDied","Data":"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"}
Jul 27 20:45:05.431292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.431178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs" event={"ID":"bf01b1a5-1ce0-422c-9a92-e28b91dc5617","Type":"ContainerDied","Data":"e51bf2b0cbb363afff6a81fbff0218d800438cf65015eff5e0547b369a930c64"}
Jul 27 20:45:05.431292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.431211    2560 scope.go:117] "RemoveContainer" containerID="1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"
Jul 27 20:45:05.432424 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.432391    2560 generic.go:356] "Generic (PLEG): container finished" podID="7e749a24-7271-4e99-b49f-eafbc0afbfce" containerID="6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766" exitCode=0
Jul 27 20:45:05.432525 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.432434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" event={"ID":"7e749a24-7271-4e99-b49f-eafbc0afbfce","Type":"ContainerDied","Data":"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"}
Jul 27 20:45:05.432525 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.432472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4" event={"ID":"7e749a24-7271-4e99-b49f-eafbc0afbfce","Type":"ContainerDied","Data":"7da50ea5b0b7365166c487b6d4d7068b376e9d6ab996706617b2f1a8fd2f4e4d"}
Jul 27 20:45:05.432525 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.432438    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"
Jul 27 20:45:05.440793 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.440770    2560 scope.go:117] "RemoveContainer" containerID="1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"
Jul 27 20:45:05.441088 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:05.441067    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb\": container with ID starting with 1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb not found: ID does not exist" containerID="1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"
Jul 27 20:45:05.441146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.441095    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb"} err="failed to get container status \"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb\": rpc error: code = NotFound desc = could not find container \"1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb\": container with ID starting with 1750f8febc6130931883ea8a7faf21b05da183571f6ebf90d123b17161af2abb not found: ID does not exist"
Jul 27 20:45:05.441146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.441112    2560 scope.go:117] "RemoveContainer" containerID="6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"
Jul 27 20:45:05.449317 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.449300    2560 scope.go:117] "RemoveContainer" containerID="6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"
Jul 27 20:45:05.449562 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:05.449540    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766\": container with ID starting with 6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766 not found: ID does not exist" containerID="6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"
Jul 27 20:45:05.449636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.449572    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766"} err="failed to get container status \"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766\": rpc error: code = NotFound desc = could not find container \"6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766\": container with ID starting with 6d9374dd4bfb1c749c8ffa97cb3e5abebdc3963feba17abec9c9af39ad22c766 not found: ID does not exist"
Jul 27 20:45:05.456186 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.456136    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:45:05.459874 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.459854    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-584365a2-5d78b7b64d-rvdxs"]
Jul 27 20:45:05.469443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.469417    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:45:05.472632 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.472613    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-584365a2-56bc688ff6-kf6r4"]
Jul 27 20:45:05.503102 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.503078    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:05.503102 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.503099    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trbbn\" (UniqueName: \"kubernetes.io/projected/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-kube-api-access-trbbn\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:05.503229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.503114    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf01b1a5-1ce0-422c-9a92-e28b91dc5617-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:05.503229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.503130    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e749a24-7271-4e99-b49f-eafbc0afbfce-plugins-config-volume\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:05.503229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.503143    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2q7d\" (UniqueName: \"kubernetes.io/projected/7e749a24-7271-4e99-b49f-eafbc0afbfce-kube-api-access-r2q7d\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:05.727125 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.727040    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e749a24-7271-4e99-b49f-eafbc0afbfce" path="/var/lib/kubelet/pods/7e749a24-7271-4e99-b49f-eafbc0afbfce/volumes"
Jul 27 20:45:05.727514 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:05.727493    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" path="/var/lib/kubelet/pods/bf01b1a5-1ce0-422c-9a92-e28b91dc5617/volumes"
Jul 27 20:45:09.529589 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:09.529547    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"]
Jul 27 20:45:09.531620 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:09.531597    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-584365a2-2x6kj"]
Jul 27 20:45:09.674200 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:09.674163    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:45:09.674456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:09.674427    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy" containerID="cri-o://93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091" gracePeriod=30
Jul 27 20:45:09.727224 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:09.727190    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f9c1ab-8c31-4d44-9b9e-46978843203f" path="/var/lib/kubelet/pods/97f9c1ab-8c31-4d44-9b9e-46978843203f/volumes"
Jul 27 20:45:10.644675 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.644633    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645027    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee42f7-106c-48a8-9b59-a00f24921b9b" containerName="cleanup"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645040    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee42f7-106c-48a8-9b59-a00f24921b9b" containerName="cleanup"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645057    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" containerName="maas-api"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645063    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" containerName="maas-api"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645071    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e749a24-7271-4e99-b49f-eafbc0afbfce" containerName="payload-pre-processing"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645076    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e749a24-7271-4e99-b49f-eafbc0afbfce" containerName="payload-pre-processing"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645147    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbee42f7-106c-48a8-9b59-a00f24921b9b" containerName="cleanup"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645154    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf01b1a5-1ce0-422c-9a92-e28b91dc5617" containerName="maas-api"
Jul 27 20:45:10.645155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.645162    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e749a24-7271-4e99-b49f-eafbc0afbfce" containerName="payload-pre-processing"
Jul 27 20:45:10.650016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.649997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.652683 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.652662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-944cd4c8-gw-openshift-default-dockercfg-tm56c\""
Jul 27 20:45:10.659924 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.659902    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:10.750115 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750115 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.750582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.750492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2ht\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.851613 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.851613 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.851855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2ht\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.851855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.851855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.851992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.852067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.852139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.852214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.852260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.852595 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.852576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.854118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.854095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.854417 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.854394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.865656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.865626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.866436 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.866416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2ht\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht\") pod \"e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") " pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:10.961714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:10.961629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:11.108866 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.108841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:11.111159 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:45:11.111127    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311d984c_0a38_4eb1_9017_b689e6c222ed.slice/crio-06add3671be6d50584a57a1bafcd77656a84444273fd6d374db17b5de27c8451 WatchSource:0}: Error finding container 06add3671be6d50584a57a1bafcd77656a84444273fd6d374db17b5de27c8451: Status 404 returned error can't find the container with id 06add3671be6d50584a57a1bafcd77656a84444273fd6d374db17b5de27c8451
Jul 27 20:45:11.113213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.113184    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:11.113284 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.113245    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:11.113284 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.113274    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:11.466620 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.466585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerStarted","Data":"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"}
Jul 27 20:45:11.466842 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.466644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerStarted","Data":"06add3671be6d50584a57a1bafcd77656a84444273fd6d374db17b5de27c8451"}
Jul 27 20:45:11.488623 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.488578    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podStartSLOduration=1.488562031 podStartE2EDuration="1.488562031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:11.486913724 +0000 UTC m=+1644.336680557" watchObservedRunningTime="2026-07-27 20:45:11.488562031 +0000 UTC m=+1644.338328861"
Jul 27 20:45:11.962435 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:11.962398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:12.967493 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:12.967462    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:13.007330 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.007293    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:13.010733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.010706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.013571 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.013543    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:45:13.013716 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.013628    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:45:13.013899 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.013878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-944cd4c8-serving-cert\""
Jul 27 20:45:13.014068 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.013980    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:45:13.020406 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.020380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:13.071728 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.071696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.071922 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.071752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9gh\" (UniqueName: \"kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.071922 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.071875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.173188 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.173148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.173393 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.173242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.173393 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.173288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9gh\" (UniqueName: \"kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.173393 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:13.173383    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-serving-cert: secret "maas-api-e2e-migrate-944cd4c8-serving-cert" not found
Jul 27 20:45:13.173564 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:13.173459    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls podName:e29f29bd-bdd3-4216-afa3-fd3b0a3b9673 nodeName:}" failed. No retries permitted until 2026-07-27 20:45:13.673440726 +0000 UTC m=+1646.523207535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls") pod "maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" (UID: "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673") : secret "maas-api-e2e-migrate-944cd4c8-serving-cert" not found
Jul 27 20:45:13.173839 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.173796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.181332 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.181312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9gh\" (UniqueName: \"kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.485556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.485526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:13.486531 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.486509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:13.677882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:13.677844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:13.678065 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:13.677978    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-serving-cert: secret "maas-api-e2e-migrate-944cd4c8-serving-cert" not found
Jul 27 20:45:13.678065 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:13.678048    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls podName:e29f29bd-bdd3-4216-afa3-fd3b0a3b9673 nodeName:}" failed. No retries permitted until 2026-07-27 20:45:14.678029132 +0000 UTC m=+1647.527795942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls") pod "maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" (UID: "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673") : secret "maas-api-e2e-migrate-944cd4c8-serving-cert" not found
Jul 27 20:45:14.687874 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.687841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:14.690165 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.690146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:14.824568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.824539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:14.927888 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.927860    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:45:14.975680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.975659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:14.977884 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:45:14.977856    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29f29bd_bdd3_4216_afa3_fd3b0a3b9673.slice/crio-48299fa74f6a4b84bf17f47104797c87ea2edf2f82b8f985f85259daba2685c2 WatchSource:0}: Error finding container 48299fa74f6a4b84bf17f47104797c87ea2edf2f82b8f985f85259daba2685c2: Status 404 returned error can't find the container with id 48299fa74f6a4b84bf17f47104797c87ea2edf2f82b8f985f85259daba2685c2
Jul 27 20:45:14.990097 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990076    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990125    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990149    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95xzh\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990277    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990317    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990347    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy\") pod \"57bfc49d-8788-401b-900d-c3aa51d51822\" (UID: \"57bfc49d-8788-401b-900d-c3aa51d51822\") "
Jul 27 20:45:14.990422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990368    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:14.990689 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:14.990758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990703    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:14.990827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990756    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:14.990827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.990786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:14.991040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.991009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data" (OuterVolumeSpecName: "istio-data") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:14.991116 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.991075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:14.992499 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.992472    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:45:14.992616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.992552    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token" (OuterVolumeSpecName: "istio-token") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:14.992877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.992855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:14.992970 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:14.992883    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh" (OuterVolumeSpecName: "kube-api-access-95xzh") pod "57bfc49d-8788-401b-900d-c3aa51d51822" (UID: "57bfc49d-8788-401b-900d-c3aa51d51822"). InnerVolumeSpecName "kube-api-access-95xzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:15.091652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091624    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xzh\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-kube-api-access-95xzh\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091649    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bfc49d-8788-401b-900d-c3aa51d51822-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091659    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bfc49d-8788-401b-900d-c3aa51d51822-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091667    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091675    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091684    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bfc49d-8788-401b-900d-c3aa51d51822-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.091912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.091692    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bfc49d-8788-401b-900d-c3aa51d51822-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:15.497059 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.497031    2560 generic.go:356] "Generic (PLEG): container finished" podID="57bfc49d-8788-401b-900d-c3aa51d51822" containerID="93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091" exitCode=0
Jul 27 20:45:15.497244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.497084    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"
Jul 27 20:45:15.497244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.497120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerDied","Data":"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"}
Jul 27 20:45:15.497244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.497172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm" event={"ID":"57bfc49d-8788-401b-900d-c3aa51d51822","Type":"ContainerDied","Data":"6137f3cadb1d4e5ccd843c7f00be1bb9de5f1e17931f11e3db2d87273e6f26a0"}
Jul 27 20:45:15.497244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.497189    2560 scope.go:117] "RemoveContainer" containerID="93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"
Jul 27 20:45:15.498659 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.498619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" event={"ID":"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673","Type":"ContainerStarted","Data":"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"}
Jul 27 20:45:15.498776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.498675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" event={"ID":"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673","Type":"ContainerStarted","Data":"48299fa74f6a4b84bf17f47104797c87ea2edf2f82b8f985f85259daba2685c2"}
Jul 27 20:45:15.506546 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.506450    2560 scope.go:117] "RemoveContainer" containerID="c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"
Jul 27 20:45:15.514416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.514398    2560 scope.go:117] "RemoveContainer" containerID="93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"
Jul 27 20:45:15.514674 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:15.514653    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091\": container with ID starting with 93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091 not found: ID does not exist" containerID="93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"
Jul 27 20:45:15.514762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.514682    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091"} err="failed to get container status \"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091\": rpc error: code = NotFound desc = could not find container \"93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091\": container with ID starting with 93c38d48672a8d953d9edd8acad235a1fe1a7021549e26494b7992bfa42ff091 not found: ID does not exist"
Jul 27 20:45:15.514762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.514698    2560 scope.go:117] "RemoveContainer" containerID="c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"
Jul 27 20:45:15.514962 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:15.514934    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38\": container with ID starting with c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38 not found: ID does not exist" containerID="c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"
Jul 27 20:45:15.515022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.514972    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38"} err="failed to get container status \"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38\": rpc error: code = NotFound desc = could not find container \"c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38\": container with ID starting with c15bc1121af6a479289fd7376737cce54dae258ea80abb42c458a54708f57b38 not found: ID does not exist"
Jul 27 20:45:15.520326 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.520284    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" podStartSLOduration=3.120629152 podStartE2EDuration="3.520269562s" totalImagesPullingTime="399.64041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:15.519187677 +0000 UTC m=+1648.368954509" watchObservedRunningTime="2026-07-27 20:45:15.520269562 +0000 UTC m=+1648.370036436"
Jul 27 20:45:15.537009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.536985    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:45:15.539674 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.539651    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-584365a2-openshift-default-64cc7457b8-xhbjm"]
Jul 27 20:45:15.727087 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:15.727055    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" path="/var/lib/kubelet/pods/57bfc49d-8788-401b-900d-c3aa51d51822/volumes"
Jul 27 20:45:16.505030 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:16.504997    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:22.516350 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:22.516321    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:24.546819 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.546780    2560 generic.go:356] "Generic (PLEG): container finished" podID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerID="26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30" exitCode=0
Jul 27 20:45:24.547260 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.546852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerDied","Data":"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"}
Jul 27 20:45:24.547260 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.547229    2560 scope.go:117] "RemoveContainer" containerID="26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"
Jul 27 20:45:24.548270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.548239    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:24.548391 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.548312    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:24.548391 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:24.548362    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:25.554104 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:25.554070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerStarted","Data":"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"}
Jul 27 20:45:25.962628 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:25.962575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:25.964147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:25.964115    2560 patch_prober.go:28] interesting pod/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 20:45:25.964378 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:25.964347    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 20:45:26.962909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:26.962867    2560 patch_prober.go:28] interesting pod/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 20:45:26.963422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:26.962944    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 20:45:27.963051 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:27.963017    2560 patch_prober.go:28] interesting pod/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 20:45:27.963609 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:27.963572    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 20:45:28.485865 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:28.485830    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:28.962834 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:28.962782    2560 patch_prober.go:28] interesting pod/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 20:45:28.963133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:28.963104    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 20:45:29.967237 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:29.967199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:29.968169 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:29.968151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:48.333691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.333649    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:48.334216 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.334079    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" podUID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" containerName="maas-api" containerID="cri-o://20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63" gracePeriod=30
Jul 27 20:45:48.591183 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.591117    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:48.671442 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.671408    2560 generic.go:356] "Generic (PLEG): container finished" podID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" containerID="20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63" exitCode=0
Jul 27 20:45:48.671607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.671450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"
Jul 27 20:45:48.671607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.671475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" event={"ID":"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673","Type":"ContainerDied","Data":"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"}
Jul 27 20:45:48.671607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.671524    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv" event={"ID":"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673","Type":"ContainerDied","Data":"48299fa74f6a4b84bf17f47104797c87ea2edf2f82b8f985f85259daba2685c2"}
Jul 27 20:45:48.671607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.671553    2560 scope.go:117] "RemoveContainer" containerID="20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"
Jul 27 20:45:48.680414 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.680392    2560 scope.go:117] "RemoveContainer" containerID="20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"
Jul 27 20:45:48.680671 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:48.680653    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63\": container with ID starting with 20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63 not found: ID does not exist" containerID="20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"
Jul 27 20:45:48.680725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.680681    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63"} err="failed to get container status \"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63\": rpc error: code = NotFound desc = could not find container \"20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63\": container with ID starting with 20ce7fd220625a45443335fe67f9940c8f4a5a2f00a94797d42f6c37f8362f63 not found: ID does not exist"
Jul 27 20:45:48.700068 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.700042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp9gh\" (UniqueName: \"kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh\") pod \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") "
Jul 27 20:45:48.700217 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.700134    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca\") pod \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") "
Jul 27 20:45:48.700263 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.700215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") pod \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\" (UID: \"e29f29bd-bdd3-4216-afa3-fd3b0a3b9673\") "
Jul 27 20:45:48.700488 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.700464    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" (UID: "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:48.702071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.702046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh" (OuterVolumeSpecName: "kube-api-access-lp9gh") pod "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" (UID: "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673"). InnerVolumeSpecName "kube-api-access-lp9gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:48.702174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.702155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" (UID: "e29f29bd-bdd3-4216-afa3-fd3b0a3b9673"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:45:48.801622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.801589    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:48.801622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.801617    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9gh\" (UniqueName: \"kubernetes.io/projected/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-kube-api-access-lp9gh\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:48.801622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.801629    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:48.993546 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.993516    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:48.996990 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:48.996967    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-944cd4c8-7b79d67dbb-wwnjv"]
Jul 27 20:45:49.726027 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:49.725994    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" path="/var/lib/kubelet/pods/e29f29bd-bdd3-4216-afa3-fd3b0a3b9673/volumes"
Jul 27 20:45:53.145100 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.145065    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:53.145960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.145314    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy" containerID="cri-o://594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef" gracePeriod=30
Jul 27 20:45:53.928831 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.928776    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:45:53.929172 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929158    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929172 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929173    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929185    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929191    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929199    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" containerName="maas-api"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929204    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" containerName="maas-api"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929261    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929268    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bfc49d-8788-401b-900d-c3aa51d51822" containerName="istio-proxy"
Jul 27 20:45:53.929303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.929274    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29f29bd-bdd3-4216-afa3-fd3b0a3b9673" containerName="maas-api"
Jul 27 20:45:53.933745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.933726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:53.936293 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.936275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ac3244c-openshift-default-dockercfg-lgxzr\""
Jul 27 20:45:53.943277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:53.943252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:45:54.047491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf8t7\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.047863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.047779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.148983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.148953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.148996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf8t7\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149917 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149917 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.149917 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.150013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.149915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.151348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.151328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.151579 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.151559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.156389 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.156367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.156656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.156637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf8t7\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7\") pod \"e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") " pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.244359 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.244289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:54.378268 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.378189    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:45:54.380683 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:45:54.380646    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78acd29c_63f1_4980_b90e_789ad3dd503d.slice/crio-db32df87eafdbf453e472bcd6aa55a0a26581459f6d455d04d64d8d4ffd0da2c WatchSource:0}: Error finding container db32df87eafdbf453e472bcd6aa55a0a26581459f6d455d04d64d8d4ffd0da2c: Status 404 returned error can't find the container with id db32df87eafdbf453e472bcd6aa55a0a26581459f6d455d04d64d8d4ffd0da2c
Jul 27 20:45:54.383547 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.383506    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:54.383657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.383581    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:54.383657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.383626    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:45:54.704029 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.703991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerStarted","Data":"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"}
Jul 27 20:45:54.704226 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.704048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerStarted","Data":"db32df87eafdbf453e472bcd6aa55a0a26581459f6d455d04d64d8d4ffd0da2c"}
Jul 27 20:45:54.726189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:54.726140    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podStartSLOduration=1.726127532 podStartE2EDuration="1.726127532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:54.723605026 +0000 UTC m=+1687.573371859" watchObservedRunningTime="2026-07-27 20:45:54.726127532 +0000 UTC m=+1687.575894362"
Jul 27 20:45:55.245459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:55.245418    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:55.249956 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:55.249931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:55.708558 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:55.708529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:55.709529 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:55.709508    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:45:56.137124 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.137086    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:45:56.140930 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.140908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.143854 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.143834    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:45:56.143854 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.143842    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:45:56.144016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.143851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ac3244c-serving-cert\""
Jul 27 20:45:56.144016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.143854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:45:56.149724 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.149699    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:45:56.269227 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.269192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.269741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.269236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.269741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.269346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbmk\" (UniqueName: \"kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.370053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.370016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.370223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.370075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.370223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.370148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmk\" (UniqueName: \"kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.370718 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.370697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.372682 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.372661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.383962 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.383937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbmk\" (UniqueName: \"kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk\") pod \"maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.453778 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.453715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:56.585858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.585832    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:45:56.588053 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:45:56.588027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f08554_cae3_446e_b5aa_ba377196269a.slice/crio-ecce76c169e3f0414df30f12dbd0094ee8d3a2bf6096d4925caf558556720e9e WatchSource:0}: Error finding container ecce76c169e3f0414df30f12dbd0094ee8d3a2bf6096d4925caf558556720e9e: Status 404 returned error can't find the container with id ecce76c169e3f0414df30f12dbd0094ee8d3a2bf6096d4925caf558556720e9e
Jul 27 20:45:56.715505 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:56.715422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" event={"ID":"51f08554-cae3-446e-b5aa-ba377196269a","Type":"ContainerStarted","Data":"ecce76c169e3f0414df30f12dbd0094ee8d3a2bf6096d4925caf558556720e9e"}
Jul 27 20:45:57.726417 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:57.726385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" event={"ID":"51f08554-cae3-446e-b5aa-ba377196269a","Type":"ContainerStarted","Data":"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"}
Jul 27 20:45:57.726787 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:57.726440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:45:57.739474 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:57.739428    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" podStartSLOduration=1.301057113 podStartE2EDuration="1.739412703s" totalImagesPullingTime="438.35559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:57.738188014 +0000 UTC m=+1690.587954843" watchObservedRunningTime="2026-07-27 20:45:57.739412703 +0000 UTC m=+1690.589179534"
Jul 27 20:45:58.397365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.397341    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:58.491890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.491777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.491890 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.491879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.491908    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.491945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw2ht\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.491972    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492002    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492066    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket\") pod \"311d984c-0a38-4eb1-9017-b689e6c222ed\" (UID: \"311d984c-0a38-4eb1-9017-b689e6c222ed\") "
Jul 27 20:45:58.492446 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492326    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:58.492446 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:58.492446 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492415    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.492619 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492598    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:58.492798 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:58.492963 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.492945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:58.494314 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.494289    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht" (OuterVolumeSpecName: "kube-api-access-cw2ht") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "kube-api-access-cw2ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:58.494418 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.494359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:45:58.494587 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.494561    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:45:58.494677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.494565    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "311d984c-0a38-4eb1-9017-b689e6c222ed" (UID: "311d984c-0a38-4eb1-9017-b689e6c222ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:58.593174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593147    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593171    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/311d984c-0a38-4eb1-9017-b689e6c222ed-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593181    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2ht\" (UniqueName: \"kubernetes.io/projected/311d984c-0a38-4eb1-9017-b689e6c222ed-kube-api-access-cw2ht\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593190    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593199    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593206    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593215    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.593353 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.593223    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/311d984c-0a38-4eb1-9017-b689e6c222ed-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:45:58.728602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.728570    2560 generic.go:356] "Generic (PLEG): container finished" podID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerID="594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef" exitCode=0
Jul 27 20:45:58.729057 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.728613    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"
Jul 27 20:45:58.729057 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.728648    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerDied","Data":"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"}
Jul 27 20:45:58.729057 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.728696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt" event={"ID":"311d984c-0a38-4eb1-9017-b689e6c222ed","Type":"ContainerDied","Data":"06add3671be6d50584a57a1bafcd77656a84444273fd6d374db17b5de27c8451"}
Jul 27 20:45:58.729057 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.728718    2560 scope.go:117] "RemoveContainer" containerID="594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"
Jul 27 20:45:58.738524 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.738501    2560 scope.go:117] "RemoveContainer" containerID="26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"
Jul 27 20:45:58.746666 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.746651    2560 scope.go:117] "RemoveContainer" containerID="594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"
Jul 27 20:45:58.746921 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:58.746904    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef\": container with ID starting with 594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef not found: ID does not exist" containerID="594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"
Jul 27 20:45:58.746977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.746929    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef"} err="failed to get container status \"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef\": rpc error: code = NotFound desc = could not find container \"594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef\": container with ID starting with 594a4c87ebea7c706de9f857f9a4e9b3dbc754bec55ff0cb6d301a4b8a8fdaef not found: ID does not exist"
Jul 27 20:45:58.746977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.746943    2560 scope.go:117] "RemoveContainer" containerID="26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"
Jul 27 20:45:58.747187 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:45:58.747169    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30\": container with ID starting with 26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30 not found: ID does not exist" containerID="26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"
Jul 27 20:45:58.747235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.747191    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30"} err="failed to get container status \"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30\": rpc error: code = NotFound desc = could not find container \"26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30\": container with ID starting with 26be0e81105a86f99971956428504b3a5f7dc3090d32c0168f6c844a82006f30 not found: ID does not exist"
Jul 27 20:45:58.751673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.751653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:58.756082 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:58.756060    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-944cd4c8-gw-openshift-default-6c97dfb6cb-mzlvt"]
Jul 27 20:45:59.725967 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:45:59.725932    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" path="/var/lib/kubelet/pods/311d984c-0a38-4eb1-9017-b689e6c222ed/volumes"
Jul 27 20:46:03.734215 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:03.734188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:46:10.710628 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.710591    2560 patch_prober.go:28] interesting pod/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 27 20:46:10.711155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.710666    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 27 20:46:10.793757 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.793728    2560 generic.go:356] "Generic (PLEG): container finished" podID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerID="46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d" exitCode=0
Jul 27 20:46:10.793954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.793827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerDied","Data":"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"}
Jul 27 20:46:10.794270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.794253    2560 scope.go:117] "RemoveContainer" containerID="46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"
Jul 27 20:46:10.795290 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.795261    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:10.795395 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.795366    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:10.795460 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:10.795417    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:11.800335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:11.800298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerStarted","Data":"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"}
Jul 27 20:46:12.244895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:12.244855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:12.245072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:12.244913    2560 patch_prober.go:28] interesting pod/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 27 20:46:12.245072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:12.244965    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 27 20:46:13.244909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:13.244874    2560 patch_prober.go:28] interesting pod/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 27 20:46:13.245309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:13.244929    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 27 20:46:14.244902 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:14.244867    2560 patch_prober.go:28] interesting pod/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 27 20:46:14.245081 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:14.244926    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 27 20:46:15.245212 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:15.245175    2560 patch_prober.go:28] interesting pod/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 27 20:46:15.245605 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:15.245231    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 27 20:46:16.248147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:16.248119    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:16.248491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:16.248458    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:16.249054 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:16.249037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:32.323572 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.323538    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:46:32.324145 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.323758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" podUID="51f08554-cae3-446e-b5aa-ba377196269a" containerName="maas-api" containerID="cri-o://0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e" gracePeriod=30
Jul 27 20:46:32.570915 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.570884    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:46:32.692692 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.692664    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls\") pod \"51f08554-cae3-446e-b5aa-ba377196269a\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") "
Jul 27 20:46:32.692878 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.692780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cbmk\" (UniqueName: \"kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk\") pod \"51f08554-cae3-446e-b5aa-ba377196269a\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") "
Jul 27 20:46:32.692878 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.692860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca\") pod \"51f08554-cae3-446e-b5aa-ba377196269a\" (UID: \"51f08554-cae3-446e-b5aa-ba377196269a\") "
Jul 27 20:46:32.693189 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.693160    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51f08554-cae3-446e-b5aa-ba377196269a" (UID: "51f08554-cae3-446e-b5aa-ba377196269a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:46:32.694677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.694651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51f08554-cae3-446e-b5aa-ba377196269a" (UID: "51f08554-cae3-446e-b5aa-ba377196269a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:46:32.694776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.694727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk" (OuterVolumeSpecName: "kube-api-access-4cbmk") pod "51f08554-cae3-446e-b5aa-ba377196269a" (UID: "51f08554-cae3-446e-b5aa-ba377196269a"). InnerVolumeSpecName "kube-api-access-4cbmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:46:32.793935 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.793907    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f08554-cae3-446e-b5aa-ba377196269a-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:32.793935 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.793932    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51f08554-cae3-446e-b5aa-ba377196269a-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:32.793935 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.793941    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbmk\" (UniqueName: \"kubernetes.io/projected/51f08554-cae3-446e-b5aa-ba377196269a-kube-api-access-4cbmk\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:32.909568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.909540    2560 generic.go:356] "Generic (PLEG): container finished" podID="51f08554-cae3-446e-b5aa-ba377196269a" containerID="0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e" exitCode=0
Jul 27 20:46:32.909726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.909583    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"
Jul 27 20:46:32.909726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.909631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" event={"ID":"51f08554-cae3-446e-b5aa-ba377196269a","Type":"ContainerDied","Data":"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"}
Jul 27 20:46:32.909726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.909679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2" event={"ID":"51f08554-cae3-446e-b5aa-ba377196269a","Type":"ContainerDied","Data":"ecce76c169e3f0414df30f12dbd0094ee8d3a2bf6096d4925caf558556720e9e"}
Jul 27 20:46:32.909726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.909702    2560 scope.go:117] "RemoveContainer" containerID="0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"
Jul 27 20:46:32.920533 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.920517    2560 scope.go:117] "RemoveContainer" containerID="0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"
Jul 27 20:46:32.920801 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:46:32.920781    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e\": container with ID starting with 0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e not found: ID does not exist" containerID="0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"
Jul 27 20:46:32.920870 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.920823    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e"} err="failed to get container status \"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e\": rpc error: code = NotFound desc = could not find container \"0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e\": container with ID starting with 0f662dfedf44adb373a6c70a2953668a15ab55240fb25acf4bedafd63bbaa23e not found: ID does not exist"
Jul 27 20:46:32.933676 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.933654    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:46:32.936623 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:32.936603    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ac3244c-cfb698859-gdmh2"]
Jul 27 20:46:33.726453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:33.726375    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f08554-cae3-446e-b5aa-ba377196269a" path="/var/lib/kubelet/pods/51f08554-cae3-446e-b5aa-ba377196269a/volumes"
Jul 27 20:46:38.045185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.045154    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:46:38.045737 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.045537    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy" containerID="cri-o://ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6" gracePeriod=30
Jul 27 20:46:38.792697 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.792659    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:46:38.793052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793037    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:38.793052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793052    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:38.793204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793072    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f08554-cae3-446e-b5aa-ba377196269a" containerName="maas-api"
Jul 27 20:46:38.793204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793078    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f08554-cae3-446e-b5aa-ba377196269a" containerName="maas-api"
Jul 27 20:46:38.793204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793141    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f08554-cae3-446e-b5aa-ba377196269a" containerName="maas-api"
Jul 27 20:46:38.793204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.793151    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:38.797546 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.797528    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.800196 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.800173    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f693d665-openshift-default-dockercfg-785lt\""
Jul 27 20:46:38.808563 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.808527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:46:38.946037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946358 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946358 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbk6l\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946431 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:38.946431 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:38.946410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbk6l\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.047954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.048229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.047957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.048229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.048056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.048357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.048337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.050090 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.050066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.050394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.050373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.055468 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.055445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbk6l\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.055691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.055671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token\") pod \"e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") " pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.108669 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.108643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:39.247548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.247523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:46:39.249595 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:46:39.249569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6d381c_42b0_4e8c_8252_d03636c47306.slice/crio-f481ce46491f6fc5e761cc3b8b8650915b40f154cbc005e3960cc73995252d1f WatchSource:0}: Error finding container f481ce46491f6fc5e761cc3b8b8650915b40f154cbc005e3960cc73995252d1f: Status 404 returned error can't find the container with id f481ce46491f6fc5e761cc3b8b8650915b40f154cbc005e3960cc73995252d1f
Jul 27 20:46:39.251719 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.251678    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:39.251840 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.251794    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:39.251894 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.251879    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:39.946042 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.946010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerStarted","Data":"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"}
Jul 27 20:46:39.946227 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.946060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerStarted","Data":"f481ce46491f6fc5e761cc3b8b8650915b40f154cbc005e3960cc73995252d1f"}
Jul 27 20:46:39.966843 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:39.966775    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podStartSLOduration=1.9667615440000001 podStartE2EDuration="1.966761544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:46:39.964933524 +0000 UTC m=+1732.814700359" watchObservedRunningTime="2026-07-27 20:46:39.966761544 +0000 UTC m=+1732.816528440"
Jul 27 20:46:40.109874 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:40.109840    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:40.114512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:40.114488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:40.950978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:40.950950    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:40.952040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:40.952020    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:41.532011 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.531978    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:46:41.532424 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.532409    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:41.532492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.532427    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:41.532492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.532481    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="311d984c-0a38-4eb1-9017-b689e6c222ed" containerName="istio-proxy"
Jul 27 20:46:41.535681 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.535665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.538339 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.538305    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f693d665-serving-cert\""
Jul 27 20:46:41.538453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.538374    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:46:41.538453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.538436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:46:41.539378 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.539362    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:46:41.544110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.543614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:46:41.672245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.672212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.672434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.672252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.672434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.672275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qngx\" (UniqueName: \"kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.772925 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.772894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.773114 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.772935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.773114 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.772960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qngx\" (UniqueName: \"kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.773114 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:46:41.773087    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-serving-cert: secret "maas-api-e2e-derive-f693d665-serving-cert" not found
Jul 27 20:46:41.773296 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:46:41.773174    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls podName:2837e77e-9970-4c35-bc48-909c9cf20c16 nodeName:}" failed. No retries permitted until 2026-07-27 20:46:42.273150937 +0000 UTC m=+1735.122917745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls") pod "maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" (UID: "2837e77e-9970-4c35-bc48-909c9cf20c16") : secret "maas-api-e2e-derive-f693d665-serving-cert" not found
Jul 27 20:46:41.773616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.773598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:41.781711 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:41.781689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qngx\" (UniqueName: \"kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:42.277235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:42.277205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:42.279657 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:42.279636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") pod \"maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:42.448335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:42.448301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:42.586355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:42.586333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:46:42.588053 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:46:42.588025    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2837e77e_9970_4c35_bc48_909c9cf20c16.slice/crio-a11bbb79a2fd09bf6d494fe3da099ec8a1df4604af93a85393d7020637a4e3e7 WatchSource:0}: Error finding container a11bbb79a2fd09bf6d494fe3da099ec8a1df4604af93a85393d7020637a4e3e7: Status 404 returned error can't find the container with id a11bbb79a2fd09bf6d494fe3da099ec8a1df4604af93a85393d7020637a4e3e7
Jul 27 20:46:42.960776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:42.960745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" event={"ID":"2837e77e-9970-4c35-bc48-909c9cf20c16","Type":"ContainerStarted","Data":"a11bbb79a2fd09bf6d494fe3da099ec8a1df4604af93a85393d7020637a4e3e7"}
Jul 27 20:46:43.303827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.303789    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:43.488313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488234    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488305    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488380    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488411    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf8t7\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:46:43.488598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data\") pod \"78acd29c-63f1-4980-b90e-789ad3dd503d\" (UID: \"78acd29c-63f1-4980-b90e-789ad3dd503d\") "
Jul 27 20:46:43.489003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488611    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:46:43.489003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488756    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:46:43.489003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:46:43.489003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.488991    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.489185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.489011    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.489185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.489026    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.489185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.489040    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78acd29c-63f1-4980-b90e-789ad3dd503d-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.489777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.489744    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data" (OuterVolumeSpecName: "istio-data") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:46:43.490720 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.490699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:46:43.490841 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.490787    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token" (OuterVolumeSpecName: "istio-token") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:46:43.490968 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.490946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:46:43.491025 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.491001    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7" (OuterVolumeSpecName: "kube-api-access-mf8t7") pod "78acd29c-63f1-4980-b90e-789ad3dd503d" (UID: "78acd29c-63f1-4980-b90e-789ad3dd503d"). InnerVolumeSpecName "kube-api-access-mf8t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:46:43.589826 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.589787    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.590163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.589834    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.590163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.589844    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf8t7\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-kube-api-access-mf8t7\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.590163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.589854    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.590163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.589862    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78acd29c-63f1-4980-b90e-789ad3dd503d-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:46:43.969259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.969223    2560 generic.go:356] "Generic (PLEG): container finished" podID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerID="ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6" exitCode=0
Jul 27 20:46:43.969421 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.969276    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"
Jul 27 20:46:43.969421 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.969303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerDied","Data":"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"}
Jul 27 20:46:43.969421 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.969353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t" event={"ID":"78acd29c-63f1-4980-b90e-789ad3dd503d","Type":"ContainerDied","Data":"db32df87eafdbf453e472bcd6aa55a0a26581459f6d455d04d64d8d4ffd0da2c"}
Jul 27 20:46:43.969421 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.969376    2560 scope.go:117] "RemoveContainer" containerID="ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"
Jul 27 20:46:43.971088 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.971056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" event={"ID":"2837e77e-9970-4c35-bc48-909c9cf20c16","Type":"ContainerStarted","Data":"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"}
Jul 27 20:46:43.971246 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.971227    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:43.978254 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.978239    2560 scope.go:117] "RemoveContainer" containerID="46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"
Jul 27 20:46:43.986098 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.986081    2560 scope.go:117] "RemoveContainer" containerID="ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"
Jul 27 20:46:43.986371 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:46:43.986353    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6\": container with ID starting with ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6 not found: ID does not exist" containerID="ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"
Jul 27 20:46:43.986430 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.986380    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6"} err="failed to get container status \"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6\": rpc error: code = NotFound desc = could not find container \"ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6\": container with ID starting with ad5c1a2b6c40564e14abeb6013e9578fc9b3abc5b4f2b20cfdced6e2d77279f6 not found: ID does not exist"
Jul 27 20:46:43.986430 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.986398    2560 scope.go:117] "RemoveContainer" containerID="46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"
Jul 27 20:46:43.986633 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:46:43.986615    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d\": container with ID starting with 46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d not found: ID does not exist" containerID="46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"
Jul 27 20:46:43.986685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.986639    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d"} err="failed to get container status \"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d\": rpc error: code = NotFound desc = could not find container \"46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d\": container with ID starting with 46a70c31b5ba9fe2626ef2f34efed17baeb4d81a857bd3ccc2c830acac0aba6d not found: ID does not exist"
Jul 27 20:46:43.987584 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.987568    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:46:43.993181 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:43.993158    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ac3244c-openshift-default-7fdd89885d-zvr5t"]
Jul 27 20:46:44.006849 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:44.006775    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" podStartSLOduration=2.412950151 podStartE2EDuration="3.006758487s" totalImagesPullingTime="593.808336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:46:44.005566816 +0000 UTC m=+1736.855333648" watchObservedRunningTime="2026-07-27 20:46:44.006758487 +0000 UTC m=+1736.856525319"
Jul 27 20:46:45.731598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:45.731564    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" path="/var/lib/kubelet/pods/78acd29c-63f1-4980-b90e-789ad3dd503d/volumes"
Jul 27 20:46:49.990272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:49.990241    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:46:55.953339 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:55.953306    2560 patch_prober.go:28] interesting pod/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 27 20:46:55.953690 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:55.953371    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 27 20:46:56.030176 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.030142    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerID="1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281" exitCode=0
Jul 27 20:46:56.030309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.030211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerDied","Data":"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"}
Jul 27 20:46:56.030655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.030637    2560 scope.go:117] "RemoveContainer" containerID="1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"
Jul 27 20:46:56.031691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.031662    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:56.031802 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.031747    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:56.031802 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:56.031795    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:46:57.036975 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:57.036942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerStarted","Data":"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"}
Jul 27 20:46:57.109205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:57.109166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:46:57.109394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:57.109376    2560 patch_prober.go:28] interesting pod/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 27 20:46:57.109439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:57.109419    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 27 20:46:58.109741 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:58.109701    2560 patch_prober.go:28] interesting pod/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 27 20:46:58.110156 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:58.109768    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 27 20:46:59.109843 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:59.109791    2560 patch_prober.go:28] interesting pod/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 27 20:46:59.110233 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:46:59.109873    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 27 20:47:00.109868 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:00.109832    2560 patch_prober.go:28] interesting pod/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 27 20:47:00.110244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:00.109897    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 27 20:47:01.113045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:01.113013    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:47:01.113410 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:01.113389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:47:01.113980 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:01.113962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:47:06.202472 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202436    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"]
Jul 27 20:47:06.202877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202794    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.202877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202803    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.202877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202838    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.202877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202847    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.203017 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202909    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.203017 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.202919    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78acd29c-63f1-4980-b90e-789ad3dd503d" containerName="istio-proxy"
Jul 27 20:47:06.206394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.206376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:06.208895 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.208871    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:47:06.213908 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.213868    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"]
Jul 27 20:47:06.291949 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.291915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f693d665-klwnj\" (UID: \"3f1aad0d-2198-491b-8b6a-5cb74074a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:06.393338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.393304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f693d665-klwnj\" (UID: \"3f1aad0d-2198-491b-8b6a-5cb74074a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:06.393932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.393913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f693d665-klwnj\" (UID: \"3f1aad0d-2198-491b-8b6a-5cb74074a03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:06.518501 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.518426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:06.851776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:06.851752    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"]
Jul 27 20:47:06.853789 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:47:06.853760    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1aad0d_2198_491b_8b6a_5cb74074a03b.slice/crio-67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed WatchSource:0}: Error finding container 67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed: Status 404 returned error can't find the container with id 67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed
Jul 27 20:47:07.085563 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:07.085526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj" event={"ID":"3f1aad0d-2198-491b-8b6a-5cb74074a03b","Type":"ContainerStarted","Data":"67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed"}
Jul 27 20:47:08.091256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:08.091220    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f1aad0d-2198-491b-8b6a-5cb74074a03b" containerID="86e1dced680540b57aa72a339d58c9dd32614522ed52a41855b44f930f384f48" exitCode=0
Jul 27 20:47:08.091680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:08.091282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj" event={"ID":"3f1aad0d-2198-491b-8b6a-5cb74074a03b","Type":"ContainerDied","Data":"86e1dced680540b57aa72a339d58c9dd32614522ed52a41855b44f930f384f48"}
Jul 27 20:47:09.224335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:09.224308    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:09.318614 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:09.318586    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca\") pod \"3f1aad0d-2198-491b-8b6a-5cb74074a03b\" (UID: \"3f1aad0d-2198-491b-8b6a-5cb74074a03b\") "
Jul 27 20:47:09.318918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:09.318895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f1aad0d-2198-491b-8b6a-5cb74074a03b" (UID: "3f1aad0d-2198-491b-8b6a-5cb74074a03b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:47:09.419761 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:09.419737    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1aad0d-2198-491b-8b6a-5cb74074a03b-maas-api-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:10.101557 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:10.101468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj" event={"ID":"3f1aad0d-2198-491b-8b6a-5cb74074a03b","Type":"ContainerDied","Data":"67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed"}
Jul 27 20:47:10.101557 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:10.101487    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"
Jul 27 20:47:10.101557 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:10.101522    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a90d281808c7d80d164ccf0619145caad6d3b914e94c228b8473a5c3f580ed"
Jul 27 20:47:16.707994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:16.707961    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:47:16.708482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:16.708228    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" podUID="2837e77e-9970-4c35-bc48-909c9cf20c16" containerName="maas-api" containerID="cri-o://6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c" gracePeriod=30
Jul 27 20:47:16.957324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:16.957299    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:47:17.082582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.082493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") pod \"2837e77e-9970-4c35-bc48-909c9cf20c16\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") "
Jul 27 20:47:17.082582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.082567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca\") pod \"2837e77e-9970-4c35-bc48-909c9cf20c16\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") "
Jul 27 20:47:17.082804 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.082673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qngx\" (UniqueName: \"kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx\") pod \"2837e77e-9970-4c35-bc48-909c9cf20c16\" (UID: \"2837e77e-9970-4c35-bc48-909c9cf20c16\") "
Jul 27 20:47:17.083026 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.082999    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2837e77e-9970-4c35-bc48-909c9cf20c16" (UID: "2837e77e-9970-4c35-bc48-909c9cf20c16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:47:17.084648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.084605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2837e77e-9970-4c35-bc48-909c9cf20c16" (UID: "2837e77e-9970-4c35-bc48-909c9cf20c16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:47:17.085021 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.084989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx" (OuterVolumeSpecName: "kube-api-access-8qngx") pod "2837e77e-9970-4c35-bc48-909c9cf20c16" (UID: "2837e77e-9970-4c35-bc48-909c9cf20c16"). InnerVolumeSpecName "kube-api-access-8qngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:47:17.139467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.139431    2560 generic.go:356] "Generic (PLEG): container finished" podID="2837e77e-9970-4c35-bc48-909c9cf20c16" containerID="6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c" exitCode=0
Jul 27 20:47:17.139618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.139477    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"
Jul 27 20:47:17.139618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.139480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" event={"ID":"2837e77e-9970-4c35-bc48-909c9cf20c16","Type":"ContainerDied","Data":"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"}
Jul 27 20:47:17.139618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.139592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs" event={"ID":"2837e77e-9970-4c35-bc48-909c9cf20c16","Type":"ContainerDied","Data":"a11bbb79a2fd09bf6d494fe3da099ec8a1df4604af93a85393d7020637a4e3e7"}
Jul 27 20:47:17.139773 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.139621    2560 scope.go:117] "RemoveContainer" containerID="6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"
Jul 27 20:47:17.148651 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.148633    2560 scope.go:117] "RemoveContainer" containerID="6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"
Jul 27 20:47:17.148932 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:47:17.148913    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c\": container with ID starting with 6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c not found: ID does not exist" containerID="6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"
Jul 27 20:47:17.148992 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.148941    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c"} err="failed to get container status \"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c\": rpc error: code = NotFound desc = could not find container \"6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c\": container with ID starting with 6f53f8af5fa7869cb2ad7d91ff0877f9544fc03c05a027846f0c0b9c9fe0d74c not found: ID does not exist"
Jul 27 20:47:17.160605 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.160583    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:47:17.164300 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.164276    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f693d665-6f6d9b47d8-qdrvs"]
Jul 27 20:47:17.183942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.183919    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qngx\" (UniqueName: \"kubernetes.io/projected/2837e77e-9970-4c35-bc48-909c9cf20c16-kube-api-access-8qngx\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:17.183942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.183941    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2837e77e-9970-4c35-bc48-909c9cf20c16-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:17.184065 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.183951    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2837e77e-9970-4c35-bc48-909c9cf20c16-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:17.727072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:17.727037    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2837e77e-9970-4c35-bc48-909c9cf20c16" path="/var/lib/kubelet/pods/2837e77e-9970-4c35-bc48-909c9cf20c16/volumes"
Jul 27 20:47:21.274204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:21.274167    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"]
Jul 27 20:47:21.276501 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:21.276477    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f693d665-klwnj"]
Jul 27 20:47:21.402243 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:21.402208    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:47:21.402541 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:21.402515    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy" containerID="cri-o://ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179" gracePeriod=30
Jul 27 20:47:21.726408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:21.726373    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f1aad0d-2198-491b-8b6a-5cb74074a03b" path="/var/lib/kubelet/pods/3f1aad0d-2198-491b-8b6a-5cb74074a03b/volumes"
Jul 27 20:47:26.656002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.655977    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:47:26.764299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764261    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764299 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764279    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764309    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbk6l\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764385    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764538    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data\") pod \"1d6d381c-42b0-4e8c-8252-d03636c47306\" (UID: \"1d6d381c-42b0-4e8c-8252-d03636c47306\") "
Jul 27 20:47:26.764944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764597    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:47:26.764944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:47:26.764944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764846    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:47:26.764944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764872    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.764944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.764865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:47:26.765606 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.765580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:47:26.766736 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.766705    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:47:26.767050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.767024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:47:26.767050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.767036    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:47:26.767185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.767066    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l" (OuterVolumeSpecName: "kube-api-access-vbk6l") pod "1d6d381c-42b0-4e8c-8252-d03636c47306" (UID: "1d6d381c-42b0-4e8c-8252-d03636c47306"). InnerVolumeSpecName "kube-api-access-vbk6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:47:26.865780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865738    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.865780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865765    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.865780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865774    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.865780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865782    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.865780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865790    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbk6l\" (UniqueName: \"kubernetes.io/projected/1d6d381c-42b0-4e8c-8252-d03636c47306-kube-api-access-vbk6l\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.866202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865799    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.866202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865821    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d6d381c-42b0-4e8c-8252-d03636c47306-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:26.866202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:26.865830    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d6d381c-42b0-4e8c-8252-d03636c47306-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:47:27.191682 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.191648    2560 generic.go:356] "Generic (PLEG): container finished" podID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerID="ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179" exitCode=0
Jul 27 20:47:27.191891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.191731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerDied","Data":"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"}
Jul 27 20:47:27.192060 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.192034    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"
Jul 27 20:47:27.198382 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.192129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf" event={"ID":"1d6d381c-42b0-4e8c-8252-d03636c47306","Type":"ContainerDied","Data":"f481ce46491f6fc5e761cc3b8b8650915b40f154cbc005e3960cc73995252d1f"}
Jul 27 20:47:27.198500 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.192164    2560 scope.go:117] "RemoveContainer" containerID="ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"
Jul 27 20:47:27.207219 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.207201    2560 scope.go:117] "RemoveContainer" containerID="1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"
Jul 27 20:47:27.214721 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.214701    2560 scope.go:117] "RemoveContainer" containerID="ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"
Jul 27 20:47:27.214977 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:47:27.214956    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179\": container with ID starting with ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179 not found: ID does not exist" containerID="ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"
Jul 27 20:47:27.215048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.214983    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179"} err="failed to get container status \"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179\": rpc error: code = NotFound desc = could not find container \"ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179\": container with ID starting with ba4d4c51c9e7e0d0fc8df5224dc0e338e4243e6097ae9053fe0d5e99e6965179 not found: ID does not exist"
Jul 27 20:47:27.215048 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.215000    2560 scope.go:117] "RemoveContainer" containerID="1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"
Jul 27 20:47:27.215204 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:47:27.215188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281\": container with ID starting with 1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281 not found: ID does not exist" containerID="1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"
Jul 27 20:47:27.215244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.215211    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281"} err="failed to get container status \"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281\": rpc error: code = NotFound desc = could not find container \"1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281\": container with ID starting with 1e253ec0a44a47dfbe32328aef9231ba47ff659b9a4b074a2267ece9c805a281 not found: ID does not exist"
Jul 27 20:47:27.219291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.219271    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:47:27.223335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.223314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f693d665-openshift-default-56cddb54f6-48jlf"]
Jul 27 20:47:27.726539 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:47:27.726511    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" path="/var/lib/kubelet/pods/1d6d381c-42b0-4e8c-8252-d03636c47306/volumes"
Jul 27 20:48:57.267000 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.266968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267320    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267330    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267338    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2837e77e-9970-4c35-bc48-909c9cf20c16" containerName="maas-api"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267343    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2837e77e-9970-4c35-bc48-909c9cf20c16" containerName="maas-api"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267356    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267362    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267374    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1aad0d-2198-491b-8b6a-5cb74074a03b" containerName="revoke-keys"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267379    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1aad0d-2198-491b-8b6a-5cb74074a03b" containerName="revoke-keys"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267435    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267445    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2837e77e-9970-4c35-bc48-909c9cf20c16" containerName="maas-api"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267452    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1aad0d-2198-491b-8b6a-5cb74074a03b" containerName="revoke-keys"
Jul 27 20:48:57.267459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.267458    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6d381c-42b0-4e8c-8252-d03636c47306" containerName="istio-proxy"
Jul 27 20:48:57.270380 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.270361    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.272965 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.272942    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e6938e60-openshift-default-dockercfg-gn46c\""
Jul 27 20:48:57.273086 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.273053    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:48:57.291422 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.291399    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:48:57.331873 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.331847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.331880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxz8\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.331903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.331932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.331995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.332016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.332030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.332049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.332155 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.332066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433109 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433109 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433111    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxz8\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433357 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433684 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433684 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433684 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433803 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433725    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.433803 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.433767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.434035 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.434016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.435748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.435724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.436015 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.435995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.443055 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.443023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.443160 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.443031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxz8\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8\") pod \"e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") " pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.581363 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.581275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:57.717827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.717784    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:48:57.719571 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:48:57.719543    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode329107c_c6e0_419a_b755_ae388049f9bd.slice/crio-1dd24ef9126d74f31e74cb794463c69758e2c39a9e50702836d1664ec32bfeb3 WatchSource:0}: Error finding container 1dd24ef9126d74f31e74cb794463c69758e2c39a9e50702836d1664ec32bfeb3: Status 404 returned error can't find the container with id 1dd24ef9126d74f31e74cb794463c69758e2c39a9e50702836d1664ec32bfeb3
Jul 27 20:48:57.721939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.721898    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:48:57.722033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.721982    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:48:57.722033 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:57.722024    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:48:58.635013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:58.634979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerStarted","Data":"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"}
Jul 27 20:48:58.635432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:58.635030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerStarted","Data":"1dd24ef9126d74f31e74cb794463c69758e2c39a9e50702836d1664ec32bfeb3"}
Jul 27 20:48:58.658911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:58.658861    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podStartSLOduration=1.658845725 podStartE2EDuration="1.658845725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:48:58.657635754 +0000 UTC m=+1871.507402592" watchObservedRunningTime="2026-07-27 20:48:58.658845725 +0000 UTC m=+1871.508612555"
Jul 27 20:48:59.581909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:59.581864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:59.586707 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:59.586684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:59.639932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:59.639902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:48:59.640635 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:48:59.640620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:00.635606 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.635558    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:00.643863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.643834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.647224 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.647199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:49:00.647614 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.647596    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e6938e60-serving-cert\""
Jul 27 20:49:00.647747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.647606    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:49:00.647901 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.647877    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:49:00.654295 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.653263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:00.764089 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.764048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.764281 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.764127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4wt\" (UniqueName: \"kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.764281 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.764185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.865744 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.865712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.865928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.865768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4wt\" (UniqueName: \"kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.865928 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.865793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.866430 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.866410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.868162 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.868138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.874797 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.874777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4wt\" (UniqueName: \"kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt\") pod \"maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:00.961740 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:00.961665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:01.093232 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:01.093205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:01.094776 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:49:01.094749    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70d1d4f_90b6_47e4_8330_f42ac04df357.slice/crio-6d99409c774801379d1b44fbe98fa762b4cea7ff91cb4e5e30457fa7550a9b1e WatchSource:0}: Error finding container 6d99409c774801379d1b44fbe98fa762b4cea7ff91cb4e5e30457fa7550a9b1e: Status 404 returned error can't find the container with id 6d99409c774801379d1b44fbe98fa762b4cea7ff91cb4e5e30457fa7550a9b1e
Jul 27 20:49:01.656642 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:01.656602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" event={"ID":"e70d1d4f-90b6-47e4-8330-f42ac04df357","Type":"ContainerStarted","Data":"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"}
Jul 27 20:49:01.657130 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:01.656696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" event={"ID":"e70d1d4f-90b6-47e4-8330-f42ac04df357","Type":"ContainerStarted","Data":"6d99409c774801379d1b44fbe98fa762b4cea7ff91cb4e5e30457fa7550a9b1e"}
Jul 27 20:49:01.657130 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:01.656964    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:01.678180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:01.678124    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" podStartSLOduration=1.200922091 podStartE2EDuration="1.678104953s" totalImagesPullingTime="477.182862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:01.676790733 +0000 UTC m=+1874.526557564" watchObservedRunningTime="2026-07-27 20:49:01.678104953 +0000 UTC m=+1874.527871787"
Jul 27 20:49:07.673098 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:07.673068    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:11.922673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:11.922633    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"]
Jul 27 20:49:11.927904 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:11.927887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:11.930726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:11.930694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:49:11.932336 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:11.932301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"]
Jul 27 20:49:12.068936 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.068886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp\" (UID: \"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:12.170418 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.170383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp\" (UID: \"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:12.171255 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.171232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp\" (UID: \"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:12.242457 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.242390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:12.379589 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.379559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"]
Jul 27 20:49:12.380658 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:49:12.380632    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013a9377_6bfc_4ba5_af2b_d0b3f27e09ff.slice/crio-09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d WatchSource:0}: Error finding container 09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d: Status 404 returned error can't find the container with id 09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d
Jul 27 20:49:12.719513 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.719474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp" event={"ID":"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff","Type":"ContainerStarted","Data":"6a2481c80c94764c317c7682ebc39f09809c99ba43b4d1a0b6b4f0f1eba436be"}
Jul 27 20:49:12.719680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.719533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp" event={"ID":"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff","Type":"ContainerStarted","Data":"09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d"}
Jul 27 20:49:12.735569 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:12.735517    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp" podStartSLOduration=1.735502493 podStartE2EDuration="1.735502493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:12.733729936 +0000 UTC m=+1885.583496767" watchObservedRunningTime="2026-07-27 20:49:12.735502493 +0000 UTC m=+1885.585269357"
Jul 27 20:49:13.728738 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.728706    2560 generic.go:356] "Generic (PLEG): container finished" podID="013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" containerID="6a2481c80c94764c317c7682ebc39f09809c99ba43b4d1a0b6b4f0f1eba436be" exitCode=0
Jul 27 20:49:13.729265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.729019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp" event={"ID":"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff","Type":"ContainerDied","Data":"6a2481c80c94764c317c7682ebc39f09809c99ba43b4d1a0b6b4f0f1eba436be"}
Jul 27 20:49:13.730722 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.730699    2560 generic.go:356] "Generic (PLEG): container finished" podID="e329107c-c6e0-419a-b755-ae388049f9bd" containerID="c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a" exitCode=0
Jul 27 20:49:13.730855 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.730747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerDied","Data":"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"}
Jul 27 20:49:13.731120 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.731108    2560 scope.go:117] "RemoveContainer" containerID="c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"
Jul 27 20:49:13.732072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.732041    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:13.732135 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.732110    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:13.732172 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:13.732142    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:14.640416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:14.640381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:14.737991 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:14.737956    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerStarted","Data":"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"}
Jul 27 20:49:14.909100 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:14.909075    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:14.996287 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:14.996246    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca\") pod \"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff\" (UID: \"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff\") "
Jul 27 20:49:14.996571 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:14.996549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" (UID: "013a9377-6bfc-4ba5-af2b-d0b3f27e09ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:49:15.097889 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.097854    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff-maas-api-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:15.582456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.582415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:15.583673 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.583640    2560 patch_prober.go:28] interesting pod/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 27 20:49:15.583823 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.583703    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 27 20:49:15.743413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.743389    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"
Jul 27 20:49:15.743730 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.743432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp" event={"ID":"013a9377-6bfc-4ba5-af2b-d0b3f27e09ff","Type":"ContainerDied","Data":"09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d"}
Jul 27 20:49:15.743730 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:15.743475    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09723f99a9d6baec98f1a59fbf9251d577281f9e4bbaf53fad14205b3e54993d"
Jul 27 20:49:16.582827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:16.582762    2560 patch_prober.go:28] interesting pod/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 27 20:49:16.583126 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:16.583097    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 27 20:49:17.582193 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:17.582153    2560 patch_prober.go:28] interesting pod/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 27 20:49:17.582835 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:17.582777    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 27 20:49:18.581821 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:18.581766    2560 patch_prober.go:28] interesting pod/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 27 20:49:18.581997 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:18.581862    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 27 20:49:19.585948 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:19.585919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:19.586333 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:19.586285    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:19.587014 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:19.586993    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:22.426055 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.426016    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:22.426540 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.426322    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerName="maas-api" containerID="cri-o://0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c" gracePeriod=30
Jul 27 20:49:22.680290 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.680229    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:22.767064 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.767028    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls\") pod \"e70d1d4f-90b6-47e4-8330-f42ac04df357\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") "
Jul 27 20:49:22.767222 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.767104    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl4wt\" (UniqueName: \"kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt\") pod \"e70d1d4f-90b6-47e4-8330-f42ac04df357\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") "
Jul 27 20:49:22.767290 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.767221    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca\") pod \"e70d1d4f-90b6-47e4-8330-f42ac04df357\" (UID: \"e70d1d4f-90b6-47e4-8330-f42ac04df357\") "
Jul 27 20:49:22.767637 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.767611    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e70d1d4f-90b6-47e4-8330-f42ac04df357" (UID: "e70d1d4f-90b6-47e4-8330-f42ac04df357"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:49:22.769286 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.769262    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e70d1d4f-90b6-47e4-8330-f42ac04df357" (UID: "e70d1d4f-90b6-47e4-8330-f42ac04df357"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:49:22.769511 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.769479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt" (OuterVolumeSpecName: "kube-api-access-hl4wt") pod "e70d1d4f-90b6-47e4-8330-f42ac04df357" (UID: "e70d1d4f-90b6-47e4-8330-f42ac04df357"). InnerVolumeSpecName "kube-api-access-hl4wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:49:22.780058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.780027    2560 generic.go:356] "Generic (PLEG): container finished" podID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerID="0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c" exitCode=0
Jul 27 20:49:22.780187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.780119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" event={"ID":"e70d1d4f-90b6-47e4-8330-f42ac04df357","Type":"ContainerDied","Data":"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"}
Jul 27 20:49:22.780187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.780159    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"
Jul 27 20:49:22.780187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.780175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" event={"ID":"e70d1d4f-90b6-47e4-8330-f42ac04df357","Type":"ContainerDied","Data":"6d99409c774801379d1b44fbe98fa762b4cea7ff91cb4e5e30457fa7550a9b1e"}
Jul 27 20:49:22.780370 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.780208    2560 scope.go:117] "RemoveContainer" containerID="0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"
Jul 27 20:49:22.790732 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.790712    2560 scope.go:117] "RemoveContainer" containerID="0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"
Jul 27 20:49:22.791073 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:49:22.791054    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c\": container with ID starting with 0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c not found: ID does not exist" containerID="0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"
Jul 27 20:49:22.791149 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.791080    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c"} err="failed to get container status \"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c\": rpc error: code = NotFound desc = could not find container \"0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c\": container with ID starting with 0868b1c86e86332ada1ad609c8233437cc3b7721d790e5d3c8bc9871bbc3852c not found: ID does not exist"
Jul 27 20:49:22.807075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.805271    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:22.809302 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.809277    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr"]
Jul 27 20:49:22.868470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.868443    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e70d1d4f-90b6-47e4-8330-f42ac04df357-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:22.868470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.868468    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4wt\" (UniqueName: \"kubernetes.io/projected/e70d1d4f-90b6-47e4-8330-f42ac04df357-kube-api-access-hl4wt\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:22.868660 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:22.868484    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70d1d4f-90b6-47e4-8330-f42ac04df357-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:23.727017 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:23.726982    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" path="/var/lib/kubelet/pods/e70d1d4f-90b6-47e4-8330-f42ac04df357/volumes"
Jul 27 20:49:25.664085 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:25.664034    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6938e60-6bc45fd779-4mxsr" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.54:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:49:26.997158 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:26.997126    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"]
Jul 27 20:49:27.000223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:27.000197    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6938e60-k7rbp"]
Jul 27 20:49:27.726826 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:27.726787    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" path="/var/lib/kubelet/pods/013a9377-6bfc-4ba5-af2b-d0b3f27e09ff/volumes"
Jul 27 20:49:33.418602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:33.418565    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:49:33.419009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:33.418889    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" containerID="cri-o://750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c" gracePeriod=30
Jul 27 20:49:34.588886 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:34.588851    2560 patch_prober.go:28] interesting pod/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:49:34.589291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:34.588920    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:49:38.668311 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.668279    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:38.809566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809566 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809546    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvxz8\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809629    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.809905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.809801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert\") pod \"e329107c-c6e0-419a-b755-ae388049f9bd\" (UID: \"e329107c-c6e0-419a-b755-ae388049f9bd\") "
Jul 27 20:49:38.810204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810050    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:49:38.810204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810185    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.810204 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:49:38.810398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:49:38.810398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:49:38.810495 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.810452    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:49:38.812051 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.812029    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8" (OuterVolumeSpecName: "kube-api-access-kvxz8") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "kube-api-access-kvxz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:49:38.812164 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.812047    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:49:38.812292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.812269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:49:38.812292 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.812275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e329107c-c6e0-419a-b755-ae388049f9bd" (UID: "e329107c-c6e0-419a-b755-ae388049f9bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:49:38.863283 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.863248    2560 generic.go:356] "Generic (PLEG): container finished" podID="e329107c-c6e0-419a-b755-ae388049f9bd" containerID="750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c" exitCode=0
Jul 27 20:49:38.863482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.863300    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"
Jul 27 20:49:38.863482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.863327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerDied","Data":"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"}
Jul 27 20:49:38.863482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.863382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4" event={"ID":"e329107c-c6e0-419a-b755-ae388049f9bd","Type":"ContainerDied","Data":"1dd24ef9126d74f31e74cb794463c69758e2c39a9e50702836d1664ec32bfeb3"}
Jul 27 20:49:38.863482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.863403    2560 scope.go:117] "RemoveContainer" containerID="750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"
Jul 27 20:49:38.872403 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.872385    2560 scope.go:117] "RemoveContainer" containerID="c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"
Jul 27 20:49:38.879874 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.879858    2560 scope.go:117] "RemoveContainer" containerID="750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"
Jul 27 20:49:38.880166 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:49:38.880146    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c\": container with ID starting with 750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c not found: ID does not exist" containerID="750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"
Jul 27 20:49:38.880247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.880173    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c"} err="failed to get container status \"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c\": rpc error: code = NotFound desc = could not find container \"750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c\": container with ID starting with 750403c9459f57ee19f90ffb7682e0442f0fa727c72b8872e514391110a4428c not found: ID does not exist"
Jul 27 20:49:38.880247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.880192    2560 scope.go:117] "RemoveContainer" containerID="c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"
Jul 27 20:49:38.880413 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:49:38.880397    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a\": container with ID starting with c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a not found: ID does not exist" containerID="c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"
Jul 27 20:49:38.880458 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.880418    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a"} err="failed to get container status \"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a\": rpc error: code = NotFound desc = could not find container \"c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a\": container with ID starting with c882beebe973436ecfcdf0e44c28ad29a43ff6cdf5847ae4f7b9918d4fe0438a not found: ID does not exist"
Jul 27 20:49:38.885222 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.885201    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:49:38.895297 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.895252    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e6938e60-openshift-default-5cdf59bb4-hk7l4"]
Jul 27 20:49:38.911376 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911346    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e329107c-c6e0-419a-b755-ae388049f9bd-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911376 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911372    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911376 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911381    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e329107c-c6e0-419a-b755-ae388049f9bd-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911391    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxz8\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-kube-api-access-kvxz8\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911400    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911408    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911416    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e329107c-c6e0-419a-b755-ae388049f9bd-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:38.911549 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:38.911423    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e329107c-c6e0-419a-b755-ae388049f9bd-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:49:39.726412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:39.726376    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" path="/var/lib/kubelet/pods/e329107c-c6e0-419a-b755-ae388049f9bd/volumes"
Jul 27 20:49:52.799939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.799902    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800249    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" containerName="revoke-keys"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800260    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" containerName="revoke-keys"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800268    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800274    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800283    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800290    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800313    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerName="maas-api"
Jul 27 20:49:52.800313 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800318    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerName="maas-api"
Jul 27 20:49:52.800568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800373    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800380    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70d1d4f-90b6-47e4-8330-f42ac04df357" containerName="maas-api"
Jul 27 20:49:52.800568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800387    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e329107c-c6e0-419a-b755-ae388049f9bd" containerName="istio-proxy"
Jul 27 20:49:52.800568 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.800396    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="013a9377-6bfc-4ba5-af2b-d0b3f27e09ff" containerName="revoke-keys"
Jul 27 20:49:52.802321 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.802306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.804939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.804916    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-afbe9b95-openshift-default-dockercfg-jblpz\""
Jul 27 20:49:52.804939 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.804917    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:49:52.813548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.813521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:49:52.834128 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tmr\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834587 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.834587 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.834452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936082 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tmr\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936602 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936764 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936764 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936738    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936894 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.936951 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.936930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.938412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.938393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.938841 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.938826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.944237 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.944218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:52.944599 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:52.944579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tmr\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr\") pod \"e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") " pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:53.113108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.113037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:53.249082 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.249049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:49:53.252905 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:49:53.252868    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ce10fe_38b2_4593_b861_d4a2fa402107.slice/crio-e5c8e0f8fbffe49d9678c3c588550713d88d9332dfd71e789accce4043c51d5f WatchSource:0}: Error finding container e5c8e0f8fbffe49d9678c3c588550713d88d9332dfd71e789accce4043c51d5f: Status 404 returned error can't find the container with id e5c8e0f8fbffe49d9678c3c588550713d88d9332dfd71e789accce4043c51d5f
Jul 27 20:49:53.255071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.255040    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:53.255149 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.255103    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:53.255149 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.255132    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:49:53.938025 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.937990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerStarted","Data":"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"}
Jul 27 20:49:53.938401 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.938040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerStarted","Data":"e5c8e0f8fbffe49d9678c3c588550713d88d9332dfd71e789accce4043c51d5f"}
Jul 27 20:49:53.957525 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:53.957460    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podStartSLOduration=1.957441271 podStartE2EDuration="1.957441271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:53.956855218 +0000 UTC m=+1926.806622051" watchObservedRunningTime="2026-07-27 20:49:53.957441271 +0000 UTC m=+1926.807208103"
Jul 27 20:49:54.113484 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:54.113453    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:54.118406 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:54.118376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:54.943371 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:54.943345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:54.944229 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:54.944207    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:49:55.895156 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.895121    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:49:55.897769 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.897750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:55.900545 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.900523    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:49:55.900656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.900554    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:49:55.900656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.900581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-afbe9b95-serving-cert\""
Jul 27 20:49:55.901664 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.901647    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:49:55.908310 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.908289    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:49:55.965144 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.965106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5jv\" (UniqueName: \"kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:55.965477 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.965239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:55.965477 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:55.965312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.066555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.066523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.066727 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.066619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5jv\" (UniqueName: \"kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.066727 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.066698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.067385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.067362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.069267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.069246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.075167 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.075142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5jv\" (UniqueName: \"kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv\") pod \"maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.210031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.209954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.342625 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.342601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:49:56.344533 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:49:56.344507    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda022f11e_57a0_4fd4_81c1_0d4c9dd6a7cc.slice/crio-72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88 WatchSource:0}: Error finding container 72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88: Status 404 returned error can't find the container with id 72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88
Jul 27 20:49:56.346031 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.346009    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:49:56.955636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.955600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" event={"ID":"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc","Type":"ContainerStarted","Data":"230b8f231d30ab29785f3133e97269cfe63fe5a2fd58661d10477451a5cf4443"}
Jul 27 20:49:56.955776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.955661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" event={"ID":"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc","Type":"ContainerStarted","Data":"72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88"}
Jul 27 20:49:56.955776 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.955692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:49:56.976595 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:49:56.976543    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" podStartSLOduration=1.477607987 podStartE2EDuration="1.976523649s" totalImagesPullingTime="498.915662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:49:56.974383348 +0000 UTC m=+1929.824150181" watchObservedRunningTime="2026-07-27 20:49:56.976523649 +0000 UTC m=+1929.826290517"
Jul 27 20:50:02.966589 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:02.966555    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:50:10.107154 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.107118    2560 patch_prober.go:28] interesting pod/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:55278->10.134.0.56:15021: read: connection reset by peer" start-of-body=
Jul 27 20:50:10.107681 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.107208    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:55278->10.134.0.56:15021: read: connection reset by peer"
Jul 27 20:50:10.605198 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.605164    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:50:10.607825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.607796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.611961 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.611939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-afbe9b95\"/\"default-dockercfg-lck46\""
Jul 27 20:50:10.612092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.612013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-afbe9b95\"/\"openshift-service-ca.crt\""
Jul 27 20:50:10.612092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.611939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-afbe9b95\"/\"e2e-lifecycle-model-afbe9b95-kserve-self-signed-certs\""
Jul 27 20:50:10.612092 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.611954    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-afbe9b95\"/\"kube-root-ca.crt\""
Jul 27 20:50:10.619586 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.619562    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:50:10.700896 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.700852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.701108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.700915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.701108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.700947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nhg\" (UniqueName: \"kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.701108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.700975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.701108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.701072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.701262 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.701109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802319 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802319 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nhg\" (UniqueName: \"kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802577 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.802912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.802894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.804714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.804691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.804908 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.804893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.810722 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.810697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nhg\" (UniqueName: \"kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg\") pod \"e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") " pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:10.919656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:10.919618    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:11.026558 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.026519    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerID="e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d" exitCode=0
Jul 27 20:50:11.026708 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.026584    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerDied","Data":"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"}
Jul 27 20:50:11.027044 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.027029    2560 scope.go:117] "RemoveContainer" containerID="e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"
Jul 27 20:50:11.028338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.028306    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:50:11.028462 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.028394    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:50:11.028462 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.028433    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:50:11.057200 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:11.057173    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:50:11.061863 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:50:11.060949    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f120af0_4c23_44bd_a7e2_c52e693b8002.slice/crio-54a32e90c4a440281960e1eda98a54248f724fdd9c2678e000ceab9d9036fea1 WatchSource:0}: Error finding container 54a32e90c4a440281960e1eda98a54248f724fdd9c2678e000ceab9d9036fea1: Status 404 returned error can't find the container with id 54a32e90c4a440281960e1eda98a54248f724fdd9c2678e000ceab9d9036fea1
Jul 27 20:50:12.034108 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.034074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerStarted","Data":"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"}
Jul 27 20:50:12.035591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.035568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" event={"ID":"7f120af0-4c23-44bd-a7e2-c52e693b8002","Type":"ContainerStarted","Data":"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"}
Jul 27 20:50:12.035700 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.035600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" event={"ID":"7f120af0-4c23-44bd-a7e2-c52e693b8002","Type":"ContainerStarted","Data":"54a32e90c4a440281960e1eda98a54248f724fdd9c2678e000ceab9d9036fea1"}
Jul 27 20:50:12.078347 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.078264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" podStartSLOduration=2.078241249 podStartE2EDuration="2.078241249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:50:12.075210424 +0000 UTC m=+1944.924977256" watchObservedRunningTime="2026-07-27 20:50:12.078241249 +0000 UTC m=+1944.928008080"
Jul 27 20:50:12.114251 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.114212    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:50:12.114469 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.114444    2560 patch_prober.go:28] interesting pod/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:50:12.114557 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:12.114534    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 27 20:50:13.114241 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:13.114203    2560 patch_prober.go:28] interesting pod/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:50:13.114651 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:13.114263    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 27 20:50:14.114256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:14.114218    2560 patch_prober.go:28] interesting pod/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:50:14.114648 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:14.114278    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 27 20:50:15.113841 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:15.113790    2560 patch_prober.go:28] interesting pod/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 27 20:50:15.114008 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:15.113867    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 27 20:50:16.116850 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:16.116798    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:50:16.117235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:16.117082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:50:16.117802 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:16.117785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:50:20.920645 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:20.920591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:20.920645 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:20.920653    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:20.933088 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:20.933056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:21.097954 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:21.097919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:50:44.961438 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:44.961217    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:50:44.962898 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:44.962472    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" podUID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" containerName="maas-api" containerID="cri-o://230b8f231d30ab29785f3133e97269cfe63fe5a2fd58661d10477451a5cf4443" gracePeriod=30
Jul 27 20:50:45.210095 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.210063    2560 generic.go:356] "Generic (PLEG): container finished" podID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" containerID="230b8f231d30ab29785f3133e97269cfe63fe5a2fd58661d10477451a5cf4443" exitCode=0
Jul 27 20:50:45.210267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.210134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" event={"ID":"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc","Type":"ContainerDied","Data":"230b8f231d30ab29785f3133e97269cfe63fe5a2fd58661d10477451a5cf4443"}
Jul 27 20:50:45.210267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.210201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2" event={"ID":"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc","Type":"ContainerDied","Data":"72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88"}
Jul 27 20:50:45.210267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.210220    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72ef0203921587072c00702a96166f50683065e327a5777f30d64704a5bb6b88"
Jul 27 20:50:45.225800 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.225730    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:50:45.312898 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.312861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca\") pod \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") "
Jul 27 20:50:45.313104 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.312926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj5jv\" (UniqueName: \"kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv\") pod \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") "
Jul 27 20:50:45.313104 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.312985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls\") pod \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\" (UID: \"a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc\") "
Jul 27 20:50:45.313327 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.313284    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" (UID: "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:50:45.315123 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.315090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" (UID: "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:50:45.315235 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.315175    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv" (OuterVolumeSpecName: "kube-api-access-tj5jv") pod "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" (UID: "a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc"). InnerVolumeSpecName "kube-api-access-tj5jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:50:45.414271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.414226    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5jv\" (UniqueName: \"kubernetes.io/projected/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-kube-api-access-tj5jv\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:50:45.414271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.414266    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:50:45.414271 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:45.414280    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:50:46.214943 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:46.214901    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"
Jul 27 20:50:46.239598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:46.239556    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:50:46.243250 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:46.243215    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afbe9b95-5dd79668f4-r6gv2"]
Jul 27 20:50:47.726916 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:47.726871    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" path="/var/lib/kubelet/pods/a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc/volumes"
Jul 27 20:50:57.617736 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:50:57.617690    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:51:12.637960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.637917    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" podUID="7f120af0-4c23-44bd-a7e2-c52e693b8002" containerName="main" containerID="cri-o://502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe" gracePeriod=45
Jul 27 20:51:12.890434 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.890364    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:51:12.976436 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976472    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42nhg\" (UniqueName: \"kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg\") pod \"7f120af0-4c23-44bd-a7e2-c52e693b8002\" (UID: \"7f120af0-4c23-44bd-a7e2-c52e693b8002\") "
Jul 27 20:51:12.976858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976733    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:12.976858 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976795    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home" (OuterVolumeSpecName: "home") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:12.977023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.976940    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:12.977087 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.977021    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:12.977087 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.977041    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:12.978832 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.978774    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm" (OuterVolumeSpecName: "dshm") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:12.978945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.978887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:51:12.978945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:12.978896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg" (OuterVolumeSpecName: "kube-api-access-42nhg") pod "7f120af0-4c23-44bd-a7e2-c52e693b8002" (UID: "7f120af0-4c23-44bd-a7e2-c52e693b8002"). InnerVolumeSpecName "kube-api-access-42nhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:51:13.078132 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.078096    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:13.078132 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.078131    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nhg\" (UniqueName: \"kubernetes.io/projected/7f120af0-4c23-44bd-a7e2-c52e693b8002-kube-api-access-42nhg\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:13.078348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.078147    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f120af0-4c23-44bd-a7e2-c52e693b8002-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:13.078348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.078159    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f120af0-4c23-44bd-a7e2-c52e693b8002-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:13.358179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.358073    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f120af0-4c23-44bd-a7e2-c52e693b8002" containerID="502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe" exitCode=0
Jul 27 20:51:13.358179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.358115    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"
Jul 27 20:51:13.358179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.358126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" event={"ID":"7f120af0-4c23-44bd-a7e2-c52e693b8002","Type":"ContainerDied","Data":"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"}
Jul 27 20:51:13.358179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.358160    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl" event={"ID":"7f120af0-4c23-44bd-a7e2-c52e693b8002","Type":"ContainerDied","Data":"54a32e90c4a440281960e1eda98a54248f724fdd9c2678e000ceab9d9036fea1"}
Jul 27 20:51:13.358179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.358178    2560 scope.go:117] "RemoveContainer" containerID="502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"
Jul 27 20:51:13.368300 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.368275    2560 scope.go:117] "RemoveContainer" containerID="502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"
Jul 27 20:51:13.368651 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:51:13.368626    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe\": container with ID starting with 502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe not found: ID does not exist" containerID="502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"
Jul 27 20:51:13.368712 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.368661    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe"} err="failed to get container status \"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe\": rpc error: code = NotFound desc = could not find container \"502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe\": container with ID starting with 502f0b60b04ec3ac19f8961771c7e02d240004041010d9a774b61b2279105dfe not found: ID does not exist"
Jul 27 20:51:13.380131 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.380093    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:51:13.383911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.383878    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-afbe9b95/e2e-lifecycle-model-afbe9b95-kserve-7cfb8446d7-vvtzl"]
Jul 27 20:51:13.727071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:13.727036    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f120af0-4c23-44bd-a7e2-c52e693b8002" path="/var/lib/kubelet/pods/7f120af0-4c23-44bd-a7e2-c52e693b8002/volumes"
Jul 27 20:51:46.605076 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:46.605039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:51:46.605638 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:46.605259    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy" containerID="cri-o://2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2" gracePeriod=30
Jul 27 20:51:47.855388 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:47.855338    2560 scope.go:117] "RemoveContainer" containerID="405968615ae077b1ff5b622d63c1fb6a53d17d31fa0de4e8df37abd351071c98"
Jul 27 20:51:51.869474 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.869443    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:51:51.940684 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940638    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940881    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940908    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.940999 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.940975    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.941386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44tmr\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr\") pod \"b5ce10fe-38b2-4593-b861-d4a2fa402107\" (UID: \"b5ce10fe-38b2-4593-b861-d4a2fa402107\") "
Jul 27 20:51:51.941386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941209    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:51.941386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941229    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:51.941386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:51:51.941596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:51.941596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941477    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:51.941596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941501    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ce10fe-38b2-4593-b861-d4a2fa402107-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:51.941596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941529    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:51.941911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.941884    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:51.943223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.943191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:51:51.943335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.943264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:51:51.943550 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.943529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr" (OuterVolumeSpecName: "kube-api-access-44tmr") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "kube-api-access-44tmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:51:51.943596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:51.943541    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5ce10fe-38b2-4593-b861-d4a2fa402107" (UID: "b5ce10fe-38b2-4593-b861-d4a2fa402107"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:51:52.042072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042027    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.042072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042064    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.042072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042073    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.042072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042081    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.042364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042091    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ce10fe-38b2-4593-b861-d4a2fa402107-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.042364 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.042100    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44tmr\" (UniqueName: \"kubernetes.io/projected/b5ce10fe-38b2-4593-b861-d4a2fa402107-kube-api-access-44tmr\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:51:52.559497 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.559455    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerID="2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2" exitCode=0
Jul 27 20:51:52.559725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.559510    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"
Jul 27 20:51:52.559725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.559558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerDied","Data":"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"}
Jul 27 20:51:52.559725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.559633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv" event={"ID":"b5ce10fe-38b2-4593-b861-d4a2fa402107","Type":"ContainerDied","Data":"e5c8e0f8fbffe49d9678c3c588550713d88d9332dfd71e789accce4043c51d5f"}
Jul 27 20:51:52.559725 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.559666    2560 scope.go:117] "RemoveContainer" containerID="2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"
Jul 27 20:51:52.569680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.569652    2560 scope.go:117] "RemoveContainer" containerID="e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"
Jul 27 20:51:52.579682 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.579656    2560 scope.go:117] "RemoveContainer" containerID="2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"
Jul 27 20:51:52.580036 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:51:52.580014    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2\": container with ID starting with 2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2 not found: ID does not exist" containerID="2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"
Jul 27 20:51:52.580081 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.580049    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2"} err="failed to get container status \"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2\": rpc error: code = NotFound desc = could not find container \"2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2\": container with ID starting with 2af6b1c020105f6f2038327d4bdcbf58b75d701a3fcafa7aabf4e4c68bce86c2 not found: ID does not exist"
Jul 27 20:51:52.580081 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.580066    2560 scope.go:117] "RemoveContainer" containerID="e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"
Jul 27 20:51:52.580317 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:51:52.580293    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d\": container with ID starting with e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d not found: ID does not exist" containerID="e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"
Jul 27 20:51:52.580383 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.580322    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d"} err="failed to get container status \"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d\": rpc error: code = NotFound desc = could not find container \"e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d\": container with ID starting with e50c0336d0022748a4c835623e13074dc437e4f81147cb54b5adb3ff4bc4848d not found: ID does not exist"
Jul 27 20:51:52.587600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.587566    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:51:52.591491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:52.591460    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-afbe9b95-openshift-default-7c4d9b5b4-k2jgv"]
Jul 27 20:51:53.726748 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:51:53.726709    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" path="/var/lib/kubelet/pods/b5ce10fe-38b2-4593-b861-d4a2fa402107/volumes"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.505130    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506063    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506083    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506114    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506123    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506135    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f120af0-4c23-44bd-a7e2-c52e693b8002" containerName="main"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506143    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f120af0-4c23-44bd-a7e2-c52e693b8002" containerName="main"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506170    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" containerName="maas-api"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506179    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" containerName="maas-api"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506354    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506370    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ce10fe-38b2-4593-b861-d4a2fa402107" containerName="istio-proxy"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506380    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f120af0-4c23-44bd-a7e2-c52e693b8002" containerName="main"
Jul 27 20:52:41.508214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.506392    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a022f11e-57a0-4fd4-81c1-0d4c9dd6a7cc" containerName="maas-api"
Jul 27 20:52:41.510779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.510756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.514497 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.514469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:52:41.514633 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.514469    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c5205b-openshift-default-dockercfg-p6jxm\""
Jul 27 20:52:41.523984 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.523956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:52:41.614860 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.614797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldfr\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.614860 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.614865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.614910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.614942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.614966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.615001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.615026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615134 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.615091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.615327 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.615134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldfr\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716603 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716797 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.716889 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.716867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.717141 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.717124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.718392 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.718369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.718745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.718727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.727115 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.727086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.727301 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.727276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldfr\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr\") pod \"e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") " pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.822982 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.822888    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:41.962619 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.962593    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:52:41.965005 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:52:41.964976    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb950d1db_ffc4_4cc0_ae75_35df09fc07f2.slice/crio-1251bb4520460964207602e7cfe8cf3a0190ac10e0c4d8047c52eefd10fa5c48 WatchSource:0}: Error finding container 1251bb4520460964207602e7cfe8cf3a0190ac10e0c4d8047c52eefd10fa5c48: Status 404 returned error can't find the container with id 1251bb4520460964207602e7cfe8cf3a0190ac10e0c4d8047c52eefd10fa5c48
Jul 27 20:52:41.967156 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.967124    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:41.967256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.967187    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:41.967256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:41.967237    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:42.811483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:42.811449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerStarted","Data":"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"}
Jul 27 20:52:42.811901 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:42.811503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerStarted","Data":"1251bb4520460964207602e7cfe8cf3a0190ac10e0c4d8047c52eefd10fa5c48"}
Jul 27 20:52:42.823679 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:42.823656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:42.834832 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:42.834764    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podStartSLOduration=1.8347308180000002 podStartE2EDuration="1.834730818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:42.83319192 +0000 UTC m=+2095.682958770" watchObservedRunningTime="2026-07-27 20:52:42.834730818 +0000 UTC m=+2095.684497651"
Jul 27 20:52:43.829230 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:43.829199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:44.823170 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.823129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:44.824207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.824185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:52:44.827245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.827207    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:52:44.831727 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.831697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:44.834497 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.834466    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:52:44.834497 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.834483    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:52:44.834752 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.834466    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:52:44.834752 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.834472    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c5205b-serving-cert\""
Jul 27 20:52:44.842871 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.842123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:52:44.947769 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.947725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:44.947769 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.947775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:44.948091 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:44.947863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqzx\" (UniqueName: \"kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.048467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.048423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.048467 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.048466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.048754 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.048497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqzx\" (UniqueName: \"kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.048754 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:52:45.048591    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-serving-cert: secret "maas-api-e2e-inf-a-c5205b-serving-cert" not found
Jul 27 20:52:45.048754 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:52:45.048668    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls podName:adf2cf3e-d7cc-42e1-a5d9-f7fb47449570 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:45.548649098 +0000 UTC m=+2098.398415908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls") pod "maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" (UID: "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570") : secret "maas-api-e2e-inf-a-c5205b-serving-cert" not found
Jul 27 20:52:45.049265 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.049243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.057519 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.057474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqzx\" (UniqueName: \"kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.552974 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.552934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.555445 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.555420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.747128 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.747060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:45.900973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:45.900944    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:52:45.903654 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:52:45.903620    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf2cf3e_d7cc_42e1_a5d9_f7fb47449570.slice/crio-df21b37b81694b3d158953370e86111bab3648a2c03f801e7069630ed693eda8 WatchSource:0}: Error finding container df21b37b81694b3d158953370e86111bab3648a2c03f801e7069630ed693eda8: Status 404 returned error can't find the container with id df21b37b81694b3d158953370e86111bab3648a2c03f801e7069630ed693eda8
Jul 27 20:52:46.836198 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:46.836159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" event={"ID":"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570","Type":"ContainerStarted","Data":"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"}
Jul 27 20:52:46.836432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:46.836224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" event={"ID":"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570","Type":"ContainerStarted","Data":"df21b37b81694b3d158953370e86111bab3648a2c03f801e7069630ed693eda8"}
Jul 27 20:52:46.836432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:46.836256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:46.854259 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:46.854183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" podStartSLOduration=2.405113739 podStartE2EDuration="2.854157155s" totalImagesPullingTime="449.043416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:46.852398098 +0000 UTC m=+2099.702164930" watchObservedRunningTime="2026-07-27 20:52:46.854157155 +0000 UTC m=+2099.703923989"
Jul 27 20:52:49.833597 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.833555    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:52:49.838774 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.838741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.841375 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.841347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-704ca2-openshift-default-dockercfg-kcm7d\""
Jul 27 20:52:49.847785 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.847753    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:52:49.995655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.995655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7vr\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.995960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.995960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.995960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.995960 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.995956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.996178 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.996066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.996178 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.996093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:49.996178 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:49.996118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.096714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.096714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.096714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096752    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7vr\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097056 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.096891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.097110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.097154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.097261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.097357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.097594 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.097527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.099932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.099904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.100110 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.100092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.105291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.105260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.105582 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.105559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7vr\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr\") pod \"e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") " pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.151496 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.151439    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:50.301244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.301215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:52:50.303580 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:52:50.303551    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e2df55_9e86_4b3e_b821_4cc7c4f71fff.slice/crio-905608daff9deaaccc6dde7b575f56b1fc98f798389aa9f394e7378e06781d63 WatchSource:0}: Error finding container 905608daff9deaaccc6dde7b575f56b1fc98f798389aa9f394e7378e06781d63: Status 404 returned error can't find the container with id 905608daff9deaaccc6dde7b575f56b1fc98f798389aa9f394e7378e06781d63
Jul 27 20:52:50.305935 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.305903    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:50.306001 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.305970    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:50.306045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.306002    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:52:50.859655 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.859612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerStarted","Data":"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"}
Jul 27 20:52:50.860184 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.859673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerStarted","Data":"905608daff9deaaccc6dde7b575f56b1fc98f798389aa9f394e7378e06781d63"}
Jul 27 20:52:50.879698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:50.879637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podStartSLOduration=1.8796189 podStartE2EDuration="1.8796189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:50.877773958 +0000 UTC m=+2103.727540793" watchObservedRunningTime="2026-07-27 20:52:50.8796189 +0000 UTC m=+2103.729385730"
Jul 27 20:52:51.152303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:51.152262    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:51.157571 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:51.157539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:51.865469 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:51.865438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:51.866620 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:51.866600    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:52:52.846800 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:52.846769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:52:52.943848 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:52.943792    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:52:52.947916 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:52.947884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:52.953076 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:52.953048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-704ca2-serving-cert\""
Jul 27 20:52:52.964784 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:52.964742    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:52:53.033371 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.033320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qbx\" (UniqueName: \"kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.033599 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.033381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.033599 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.033409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.134431 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.134387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qbx\" (UniqueName: \"kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.134617 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.134447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.134617 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.134472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.134699 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:52:53.134625    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-serving-cert: secret "maas-api-e2e-inf-b-704ca2-serving-cert" not found
Jul 27 20:52:53.134731 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:52:53.134699    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls podName:4667a533-b1f8-4a45-aec6-e9db8da6f61a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:53.634680774 +0000 UTC m=+2106.484447585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls") pod "maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" (UID: "4667a533-b1f8-4a45-aec6-e9db8da6f61a") : secret "maas-api-e2e-inf-b-704ca2-serving-cert" not found
Jul 27 20:52:53.135238 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.135218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.143100 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.143067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qbx\" (UniqueName: \"kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.640378 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.640318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.648614 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.648578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") pod \"maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:53.860148 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:53.860093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:54.048220 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:54.048181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:52:54.051200 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:52:54.051160    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4667a533_b1f8_4a45_aec6_e9db8da6f61a.slice/crio-fafdeb1ca2fdbcbdc828485f092898094839dc017d5781091f30bc01c09ac661 WatchSource:0}: Error finding container fafdeb1ca2fdbcbdc828485f092898094839dc017d5781091f30bc01c09ac661: Status 404 returned error can't find the container with id fafdeb1ca2fdbcbdc828485f092898094839dc017d5781091f30bc01c09ac661
Jul 27 20:52:54.900661 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:54.900607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" event={"ID":"4667a533-b1f8-4a45-aec6-e9db8da6f61a","Type":"ContainerStarted","Data":"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"}
Jul 27 20:52:54.900856 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:54.900689    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:52:54.900856 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:54.900712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" event={"ID":"4667a533-b1f8-4a45-aec6-e9db8da6f61a","Type":"ContainerStarted","Data":"fafdeb1ca2fdbcbdc828485f092898094839dc017d5781091f30bc01c09ac661"}
Jul 27 20:52:54.920285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:52:54.920227    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" podStartSLOduration=2.554947017 podStartE2EDuration="2.920207381s" totalImagesPullingTime="365.260364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:54.918324343 +0000 UTC m=+2107.768091174" watchObservedRunningTime="2026-07-27 20:52:54.920207381 +0000 UTC m=+2107.769974212"
Jul 27 20:53:00.768164 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.767192    2560 patch_prober.go:28] interesting pod/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:37670->10.134.0.59:15021: read: connection reset by peer" start-of-body=
Jul 27 20:53:00.768164 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.767273    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:37670->10.134.0.59:15021: read: connection reset by peer"
Jul 27 20:53:00.911520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.911492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:53:00.936548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.936461    2560 generic.go:356] "Generic (PLEG): container finished" podID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerID="0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7" exitCode=0
Jul 27 20:53:00.936703 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.936557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerDied","Data":"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"}
Jul 27 20:53:00.937074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.937057    2560 scope.go:117] "RemoveContainer" containerID="0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"
Jul 27 20:53:00.938246 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.938206    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:00.938359 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.938303    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:00.938419 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:00.938355    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:01.943691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:01.943649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerStarted","Data":"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"}
Jul 27 20:53:02.824112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:02.824062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:53:02.824338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:02.824313    2560 patch_prober.go:28] interesting pod/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 27 20:53:02.824386 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:02.824365    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 27 20:53:03.823677 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:03.823640    2560 patch_prober.go:28] interesting pod/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 27 20:53:03.824222 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:03.824192    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 27 20:53:04.823505 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:04.823459    2560 patch_prober.go:28] interesting pod/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 27 20:53:04.823699 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:04.823516    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 27 20:53:05.824230 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:05.824188    2560 patch_prober.go:28] interesting pod/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 27 20:53:05.824637 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:05.824279    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 27 20:53:06.828324 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:06.828291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:53:06.828801 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:06.828697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:53:06.829373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:06.829347    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:53:07.906050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:07.905996    2560 patch_prober.go:28] interesting pod/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:53:07.906469 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:07.906069    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:53:08.985247 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.985216    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerID="330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647" exitCode=0
Jul 27 20:53:08.985732 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.985295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerDied","Data":"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"}
Jul 27 20:53:08.985777 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.985746    2560 scope.go:117] "RemoveContainer" containerID="330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"
Jul 27 20:53:08.986912 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.986883    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:08.986977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.986954    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:08.987055 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:08.986984    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:53:09.993097 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:09.993062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerStarted","Data":"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"}
Jul 27 20:53:10.152005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:10.151958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:53:10.152252 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:10.152232    2560 patch_prober.go:28] interesting pod/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 27 20:53:10.152309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:10.152277    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 27 20:53:11.152147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.152113    2560 patch_prober.go:28] interesting pod/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 27 20:53:11.152625 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.152596    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 27 20:53:11.804465 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.804290    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:53:11.809772 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.809736    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.815133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.814016    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5205b\"/\"openshift-service-ca.crt\""
Jul 27 20:53:11.815133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.815092    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5205b\"/\"kube-root-ca.crt\""
Jul 27 20:53:11.815133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.815100    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5205b\"/\"default-dockercfg-hq2xl\""
Jul 27 20:53:11.815348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.815335    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5205b\"/\"test-model-c5205b-kserve-self-signed-certs\""
Jul 27 20:53:11.816339 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.816140    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:53:11.826109 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.826288 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.826288 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.826405 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.826462 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.826462 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.826447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsv5\" (UniqueName: \"kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927540 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927759 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927759 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsv5\" (UniqueName: \"kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927759 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.927978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.927934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.928143 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.928092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.928283 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.928233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.930090 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.930064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.930508 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.930491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:11.936598 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:11.936562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsv5\" (UniqueName: \"kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5\") pod \"test-model-c5205b-kserve-6c6796f475-2phtq\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") " pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:12.124170 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:12.124136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:12.151911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:12.151870    2560 patch_prober.go:28] interesting pod/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 27 20:53:12.152116 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:12.151933    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 27 20:53:12.316009 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:12.315975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:53:12.316957 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:53:12.316933    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfefcebfc_45d1_4df3_a0e4_9aeea3c84884.slice/crio-dbd3d94bfb2513f7337a0c3a404a7d6339bcbe0ab1486a6d6d764128945a62a4 WatchSource:0}: Error finding container dbd3d94bfb2513f7337a0c3a404a7d6339bcbe0ab1486a6d6d764128945a62a4: Status 404 returned error can't find the container with id dbd3d94bfb2513f7337a0c3a404a7d6339bcbe0ab1486a6d6d764128945a62a4
Jul 27 20:53:13.010443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:13.010406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" event={"ID":"fefcebfc-45d1-4df3-a0e4-9aeea3c84884","Type":"ContainerStarted","Data":"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"}
Jul 27 20:53:13.010652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:13.010451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" event={"ID":"fefcebfc-45d1-4df3-a0e4-9aeea3c84884","Type":"ContainerStarted","Data":"dbd3d94bfb2513f7337a0c3a404a7d6339bcbe0ab1486a6d6d764128945a62a4"}
Jul 27 20:53:13.028163 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:13.028105    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" podStartSLOduration=2.02808678 podStartE2EDuration="2.02808678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:13.026702315 +0000 UTC m=+2125.876469149" watchObservedRunningTime="2026-07-27 20:53:13.02808678 +0000 UTC m=+2125.877853601"
Jul 27 20:53:13.152453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:13.152413    2560 patch_prober.go:28] interesting pod/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 27 20:53:13.152652 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:13.152473    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 27 20:53:14.155285 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:14.155251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:53:14.155782 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:14.155750    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:53:14.156419 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:14.156395    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:53:22.124765 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:22.124725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:22.125242 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:22.125025    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:22.137742 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:22.137712    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:23.078932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:23.078901    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:53:47.897981 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:53:47.897897    2560 scope.go:117] "RemoveContainer" containerID="86e1dced680540b57aa72a339d58c9dd32614522ed52a41855b44f930f384f48"
Jul 27 20:54:22.083789 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:22.083753    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:54:23.230645 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.230595    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"]
Jul 27 20:54:23.233112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.233096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:23.235872 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.235851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 20:54:23.240930 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.240275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm\" (UID: \"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:23.241456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.241431    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"]
Jul 27 20:54:23.341205 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.341168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm\" (UID: \"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:23.341837 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.341798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm\" (UID: \"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:23.545518 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.545433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:23.682308 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:23.682274    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"]
Jul 27 20:54:23.683903 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:54:23.683879    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f3e6c1_c5bd_4f3b_9427_061e2600cc7c.slice/crio-20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20 WatchSource:0}: Error finding container 20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20: Status 404 returned error can't find the container with id 20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20
Jul 27 20:54:24.398750 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:24.398712    2560 generic.go:356] "Generic (PLEG): container finished" podID="a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" containerID="56748ade7f0abb6ca8888ff48d008b431fe414fab435cc3a08d405397f19d270" exitCode=0
Jul 27 20:54:24.399124 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:24.398772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm" event={"ID":"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c","Type":"ContainerDied","Data":"56748ade7f0abb6ca8888ff48d008b431fe414fab435cc3a08d405397f19d270"}
Jul 27 20:54:24.399124 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:24.398842    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm" event={"ID":"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c","Type":"ContainerStarted","Data":"20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20"}
Jul 27 20:54:25.534947 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:25.534922    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:25.558142 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:25.558101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca\") pod \"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c\" (UID: \"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c\") "
Jul 27 20:54:25.558453 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:25.558432    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" (UID: "a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:25.558551 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:25.558537    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c-maas-api-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:26.409853 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:26.409803    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"
Jul 27 20:54:26.409853 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:26.409846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm" event={"ID":"a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c","Type":"ContainerDied","Data":"20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20"}
Jul 27 20:54:26.410071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:26.409898    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20debd1e6f0b8c4cf11707f76e8fc0fd6ca6db7235798f8a83dd8717ea3feb20"
Jul 27 20:54:33.634050 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:33.634010    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:54:33.634491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:33.634277    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" podUID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" containerName="maas-api" containerID="cri-o://40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860" gracePeriod=30
Jul 27 20:54:34.088174 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.088151    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:54:34.145195 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.145160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca\") pod \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") "
Jul 27 20:54:34.145394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.145220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") pod \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") "
Jul 27 20:54:34.145394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.145269    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kqzx\" (UniqueName: \"kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx\") pod \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\" (UID: \"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570\") "
Jul 27 20:54:34.145619 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.145595    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" (UID: "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:34.147322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.147298    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" (UID: "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:34.147444 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.147424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx" (OuterVolumeSpecName: "kube-api-access-5kqzx") pod "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" (UID: "adf2cf3e-d7cc-42e1-a5d9-f7fb47449570"). InnerVolumeSpecName "kube-api-access-5kqzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:34.246385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.246291    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:34.246385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.246326    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kqzx\" (UniqueName: \"kubernetes.io/projected/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-kube-api-access-5kqzx\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:34.246385 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.246337    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:34.449256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.449225    2560 generic.go:356] "Generic (PLEG): container finished" podID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" containerID="40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860" exitCode=0
Jul 27 20:54:34.449412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.449270    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"
Jul 27 20:54:34.449412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.449281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" event={"ID":"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570","Type":"ContainerDied","Data":"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"}
Jul 27 20:54:34.449412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.449332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c" event={"ID":"adf2cf3e-d7cc-42e1-a5d9-f7fb47449570","Type":"ContainerDied","Data":"df21b37b81694b3d158953370e86111bab3648a2c03f801e7069630ed693eda8"}
Jul 27 20:54:34.449412 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.449353    2560 scope.go:117] "RemoveContainer" containerID="40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"
Jul 27 20:54:34.463038 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.463015    2560 scope.go:117] "RemoveContainer" containerID="40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"
Jul 27 20:54:34.463365 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:54:34.463341    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860\": container with ID starting with 40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860 not found: ID does not exist" containerID="40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"
Jul 27 20:54:34.463456 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.463375    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860"} err="failed to get container status \"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860\": rpc error: code = NotFound desc = could not find container \"40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860\": container with ID starting with 40db0a7d46b80175809612722e64584cafc665a32eda1d2f09d9f702360e0860 not found: ID does not exist"
Jul 27 20:54:34.473460 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.473435    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:54:34.476822 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:34.476789    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5205b-6db6c47fbb-bbl8c"]
Jul 27 20:54:35.727354 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:35.727308    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" path="/var/lib/kubelet/pods/adf2cf3e-d7cc-42e1-a5d9-f7fb47449570/volumes"
Jul 27 20:54:37.103615 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.103575    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" podUID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" containerName="main" containerID="cri-o://a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4" gracePeriod=45
Jul 27 20:54:37.365373 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.365349    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:54:37.465187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.465149    2560 generic.go:356] "Generic (PLEG): container finished" podID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" containerID="a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4" exitCode=0
Jul 27 20:54:37.465402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.465200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"
Jul 27 20:54:37.465402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.465243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" event={"ID":"fefcebfc-45d1-4df3-a0e4-9aeea3c84884","Type":"ContainerDied","Data":"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"}
Jul 27 20:54:37.465402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.465299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq" event={"ID":"fefcebfc-45d1-4df3-a0e4-9aeea3c84884","Type":"ContainerDied","Data":"dbd3d94bfb2513f7337a0c3a404a7d6339bcbe0ab1486a6d6d764128945a62a4"}
Jul 27 20:54:37.465402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.465325    2560 scope.go:117] "RemoveContainer" containerID="a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"
Jul 27 20:54:37.474574 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.474559    2560 scope.go:117] "RemoveContainer" containerID="a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"
Jul 27 20:54:37.474863 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:54:37.474843    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4\": container with ID starting with a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4 not found: ID does not exist" containerID="a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"
Jul 27 20:54:37.474936 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.474872    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4"} err="failed to get container status \"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4\": rpc error: code = NotFound desc = could not find container \"a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4\": container with ID starting with a49af8ce8da2a809872f7d20779b94a0185dd3e52069dd051a6a84268db829f4 not found: ID does not exist"
Jul 27 20:54:37.478230 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478329 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478254    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478329 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqsv5\" (UniqueName: \"kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478402 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478393    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478438    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir\") pod \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\" (UID: \"fefcebfc-45d1-4df3-a0e4-9aeea3c84884\") "
Jul 27 20:54:37.478483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478470    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache" (OuterVolumeSpecName: "model-cache") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:37.478587 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home" (OuterVolumeSpecName: "home") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:37.478746 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478715    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:37.478877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478764    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.478877 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.478781    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.480522 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.480493    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:37.480617 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.480548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm" (OuterVolumeSpecName: "dshm") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:37.480617 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.480551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5" (OuterVolumeSpecName: "kube-api-access-zqsv5") pod "fefcebfc-45d1-4df3-a0e4-9aeea3c84884" (UID: "fefcebfc-45d1-4df3-a0e4-9aeea3c84884"). InnerVolumeSpecName "kube-api-access-zqsv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:37.580306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.580268    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.580306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.580297    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.580306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.580309    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqsv5\" (UniqueName: \"kubernetes.io/projected/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-kube-api-access-zqsv5\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.580306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.580318    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fefcebfc-45d1-4df3-a0e4-9aeea3c84884-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:37.783480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.783447    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:54:37.789700 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:37.789670    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c5205b/test-model-c5205b-kserve-6c6796f475-2phtq"]
Jul 27 20:54:38.322830 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:38.322772    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"]
Jul 27 20:54:38.324451 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:38.324427    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5205b-sdmlm"]
Jul 27 20:54:39.726319 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:39.726287    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" path="/var/lib/kubelet/pods/a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c/volumes"
Jul 27 20:54:39.726698 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:39.726684    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" path="/var/lib/kubelet/pods/fefcebfc-45d1-4df3-a0e4-9aeea3c84884/volumes"
Jul 27 20:54:44.797656 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:44.797618    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:54:44.798151 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:44.797857    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy" containerID="cri-o://943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2" gracePeriod=30
Jul 27 20:54:50.047541 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.047515    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:54:50.202863 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.202784    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.202890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.202914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.202930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203047 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203039    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203067    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jldfr\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203148    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert\") pod \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\" (UID: \"b950d1db-ffc4-4cc0-ae75-35df09fc07f2\") "
Jul 27 20:54:50.203341 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203247    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:50.203552 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:50.203552 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203512    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:50.203663 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203568    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.203663 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203589    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.203663 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203566    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:50.204005 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.203984    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:50.205103 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.205073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:50.205222 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.205162    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:54:50.205529 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.205504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:50.205636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.205619    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr" (OuterVolumeSpecName: "kube-api-access-jldfr") pod "b950d1db-ffc4-4cc0-ae75-35df09fc07f2" (UID: "b950d1db-ffc4-4cc0-ae75-35df09fc07f2"). InnerVolumeSpecName "kube-api-access-jldfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:50.304678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304641    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304672    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304681    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304678 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304689    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304696    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304705    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldfr\" (UniqueName: \"kubernetes.io/projected/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-kube-api-access-jldfr\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.304996 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.304715    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b950d1db-ffc4-4cc0-ae75-35df09fc07f2-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:50.535071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.534972    2560 generic.go:356] "Generic (PLEG): container finished" podID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerID="943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2" exitCode=0
Jul 27 20:54:50.535071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.535017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerDied","Data":"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"}
Jul 27 20:54:50.535071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.535025    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"
Jul 27 20:54:50.535071 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.535060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw" event={"ID":"b950d1db-ffc4-4cc0-ae75-35df09fc07f2","Type":"ContainerDied","Data":"1251bb4520460964207602e7cfe8cf3a0190ac10e0c4d8047c52eefd10fa5c48"}
Jul 27 20:54:50.535401 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.535082    2560 scope.go:117] "RemoveContainer" containerID="943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"
Jul 27 20:54:50.544522 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.544504    2560 scope.go:117] "RemoveContainer" containerID="0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"
Jul 27 20:54:50.552735 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.552711    2560 scope.go:117] "RemoveContainer" containerID="943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"
Jul 27 20:54:50.553007 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:54:50.552987    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2\": container with ID starting with 943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2 not found: ID does not exist" containerID="943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"
Jul 27 20:54:50.553083 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.553020    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2"} err="failed to get container status \"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2\": rpc error: code = NotFound desc = could not find container \"943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2\": container with ID starting with 943134a0c16e9de86abb807b6bb9e258ea148c29b89771240c97ba526c6b96b2 not found: ID does not exist"
Jul 27 20:54:50.553083 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.553043    2560 scope.go:117] "RemoveContainer" containerID="0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"
Jul 27 20:54:50.553278 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:54:50.553260    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7\": container with ID starting with 0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7 not found: ID does not exist" containerID="0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"
Jul 27 20:54:50.553322 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.553285    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7"} err="failed to get container status \"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7\": rpc error: code = NotFound desc = could not find container \"0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7\": container with ID starting with 0a6a4be732091e33012d7e8652d4bb67872999ae8763c177a719793525593eb7 not found: ID does not exist"
Jul 27 20:54:50.557340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.557317    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:54:50.563723 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:50.563700    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c5205b-openshift-default-69469c97d4-jtvvw"]
Jul 27 20:54:51.725991 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:51.725952    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" path="/var/lib/kubelet/pods/b950d1db-ffc4-4cc0-ae75-35df09fc07f2/volumes"
Jul 27 20:54:55.576511 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:55.576479    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:54:55.577190 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:55.576768    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerName="maas-api" containerID="cri-o://5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225" gracePeriod=30
Jul 27 20:54:55.906817 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:55.906764    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.62:8443/health\": dial tcp 10.134.0.62:8443: connect: connection refused"
Jul 27 20:54:56.031133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.031103    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:54:56.160753 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.160717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca\") pod \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") "
Jul 27 20:54:56.160973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.160794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qbx\" (UniqueName: \"kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx\") pod \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") "
Jul 27 20:54:56.160973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.160870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") pod \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\" (UID: \"4667a533-b1f8-4a45-aec6-e9db8da6f61a\") "
Jul 27 20:54:56.161187 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.161161    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4667a533-b1f8-4a45-aec6-e9db8da6f61a" (UID: "4667a533-b1f8-4a45-aec6-e9db8da6f61a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:56.162962 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.162932    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4667a533-b1f8-4a45-aec6-e9db8da6f61a" (UID: "4667a533-b1f8-4a45-aec6-e9db8da6f61a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:56.163076 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.162994    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx" (OuterVolumeSpecName: "kube-api-access-d5qbx") pod "4667a533-b1f8-4a45-aec6-e9db8da6f61a" (UID: "4667a533-b1f8-4a45-aec6-e9db8da6f61a"). InnerVolumeSpecName "kube-api-access-d5qbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:56.262441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.262395    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4667a533-b1f8-4a45-aec6-e9db8da6f61a-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:56.262441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.262433    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4667a533-b1f8-4a45-aec6-e9db8da6f61a-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:56.262663 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.262458    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qbx\" (UniqueName: \"kubernetes.io/projected/4667a533-b1f8-4a45-aec6-e9db8da6f61a-kube-api-access-d5qbx\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:54:56.566340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.566240    2560 generic.go:356] "Generic (PLEG): container finished" podID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerID="5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225" exitCode=0
Jul 27 20:54:56.566340 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.566287    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"
Jul 27 20:54:56.566535 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.566339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" event={"ID":"4667a533-b1f8-4a45-aec6-e9db8da6f61a","Type":"ContainerDied","Data":"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"}
Jul 27 20:54:56.566535 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.566401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7" event={"ID":"4667a533-b1f8-4a45-aec6-e9db8da6f61a","Type":"ContainerDied","Data":"fafdeb1ca2fdbcbdc828485f092898094839dc017d5781091f30bc01c09ac661"}
Jul 27 20:54:56.566535 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.566431    2560 scope.go:117] "RemoveContainer" containerID="5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"
Jul 27 20:54:56.577616 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.577422    2560 scope.go:117] "RemoveContainer" containerID="5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"
Jul 27 20:54:56.577885 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:54:56.577698    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225\": container with ID starting with 5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225 not found: ID does not exist" containerID="5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"
Jul 27 20:54:56.577885 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.577726    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225"} err="failed to get container status \"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225\": rpc error: code = NotFound desc = could not find container \"5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225\": container with ID starting with 5d12cfd99a80e8ecf23177a54d0f04595a31c3c1910fe804e1932fceb9c7e225 not found: ID does not exist"
Jul 27 20:54:56.588653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.588628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:54:56.595147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:56.595121    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-704ca2-566cd5cd57-mbdc7"]
Jul 27 20:54:57.727136 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:54:57.727101    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" path="/var/lib/kubelet/pods/4667a533-b1f8-4a45-aec6-e9db8da6f61a/volumes"
Jul 27 20:55:06.566045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:06.566008    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:55:06.566491 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:06.566228    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy" containerID="cri-o://090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c" gracePeriod=30
Jul 27 20:55:07.468572 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468537    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:55:07.468927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468915    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" containerName="maas-api"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468929    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" containerName="maas-api"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468945    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerName="maas-api"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468951    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerName="maas-api"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468957    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468963    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468969    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.468977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468975    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468988    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" containerName="revoke-keys"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.468993    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" containerName="revoke-keys"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469003    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" containerName="main"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469007    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" containerName="main"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469067    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4667a533-b1f8-4a45-aec6-e9db8da6f61a" containerName="maas-api"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469077    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf2cf3e-d7cc-42e1-a5d9-f7fb47449570" containerName="maas-api"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469086    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469091    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f3e6c1-c5bd-4f3b-9427-061e2600cc7c" containerName="revoke-keys"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469097    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b950d1db-ffc4-4cc0-ae75-35df09fc07f2" containerName="istio-proxy"
Jul 27 20:55:07.469207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.469103    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefcebfc-45d1-4df3-a0e4-9aeea3c84884" containerName="main"
Jul 27 20:55:07.473483 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.473462    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.476094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.476071    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cc8abd-openshift-default-dockercfg-7k4c4\""
Jul 27 20:55:07.482891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.482866    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:55:07.565074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.564947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.564988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565074 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4crp\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.565441 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.565363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.666442 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.666442 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4crp\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667037 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.666898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667526 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.667190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667526 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.667219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.667526 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.667505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.669118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.669096    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.669243 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.669219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.674685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.674660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.674845 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.674737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4crp\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp\") pod \"e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") " pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.785086 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.785049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:07.923017 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.922982    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:55:07.925611 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:55:07.925582    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd38041e_b64f_4ab6_a69d_7cec66a44889.slice/crio-3d4b7478063351cf2d380c85dcf20c017057b8c24dda47942fd57d9db1fa910b WatchSource:0}: Error finding container 3d4b7478063351cf2d380c85dcf20c017057b8c24dda47942fd57d9db1fa910b: Status 404 returned error can't find the container with id 3d4b7478063351cf2d380c85dcf20c017057b8c24dda47942fd57d9db1fa910b
Jul 27 20:55:07.927775 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.927742    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:07.927911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.927837    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:07.927911 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:07.927889    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:08.627561 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:08.627518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerStarted","Data":"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"}
Jul 27 20:55:08.627743 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:08.627607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerStarted","Data":"3d4b7478063351cf2d380c85dcf20c017057b8c24dda47942fd57d9db1fa910b"}
Jul 27 20:55:08.649461 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:08.649406    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podStartSLOduration=1.649391418 podStartE2EDuration="1.649391418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:08.647132105 +0000 UTC m=+2241.496898948" watchObservedRunningTime="2026-07-27 20:55:08.649391418 +0000 UTC m=+2241.499158249"
Jul 27 20:55:08.785744 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:08.785670    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:09.790420 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:09.790389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:10.638118 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.638073    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:10.639107 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.639085    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:10.644482 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.644456    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:10.648607 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.648589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.651087 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.651061    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:55:10.651087 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.651075    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cc8abd-serving-cert\""
Jul 27 20:55:10.651274 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.651136    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:55:10.651274 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.651166    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:55:10.659170 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.659146    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:10.695515 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.695482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.695714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.695534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plkg2\" (UniqueName: \"kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.695714 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.695590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.796518 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.796483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.797022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.796525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plkg2\" (UniqueName: \"kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.797022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.796578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.797022 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:55:10.796720    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-serving-cert: secret "maas-api-e2e-api-a-cc8abd-serving-cert" not found
Jul 27 20:55:10.797022 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:55:10.796793    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls podName:71655fca-159d-488a-b111-ba3338211a09 nodeName:}" failed. No retries permitted until 2026-07-27 20:55:11.296770506 +0000 UTC m=+2244.146537321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls") pod "maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" (UID: "71655fca-159d-488a-b111-ba3338211a09") : secret "maas-api-e2e-api-a-cc8abd-serving-cert" not found
Jul 27 20:55:10.797307 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.797141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:10.805066 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:10.805043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plkg2\" (UniqueName: \"kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:11.301348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.301305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:11.303878 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.303852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") pod \"maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:11.560767 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.560665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:11.705365 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.705332    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:11.707472 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:55:11.707442    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71655fca_159d_488a_b111_ba3338211a09.slice/crio-d9ea27f27dfd592cc01c32b0edbedbc15bc6c860708c3d13f4c008ded2eb3278 WatchSource:0}: Error finding container d9ea27f27dfd592cc01c32b0edbedbc15bc6c860708c3d13f4c008ded2eb3278: Status 404 returned error can't find the container with id d9ea27f27dfd592cc01c32b0edbedbc15bc6c860708c3d13f4c008ded2eb3278
Jul 27 20:55:11.709022 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.708992    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:55:11.829392 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.829370    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:55:11.907303 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907266    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907400    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c7vr\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907662 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907662 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907662 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907662 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907658    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket\") pod \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\" (UID: \"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff\") "
Jul 27 20:55:11.907913 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.907691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:11.908080 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.908035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:11.908080 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.908064    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:11.908398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.908246    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:11.908398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.908367    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:11.908561 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.908536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:11.909901 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.909866    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:11.910010 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.909919    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr" (OuterVolumeSpecName: "kube-api-access-4c7vr") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "kube-api-access-4c7vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:11.910072 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.910021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:11.910132 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:11.910118    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" (UID: "e6e2df55-9e86-4b3e-b821-4cc7c4f71fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:55:12.009512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009476    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009507    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009512 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009516    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009532    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009543    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009551    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009559    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.009758 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.009566    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c7vr\" (UniqueName: \"kubernetes.io/projected/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff-kube-api-access-4c7vr\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:12.648753 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.648716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" event={"ID":"71655fca-159d-488a-b111-ba3338211a09","Type":"ContainerStarted","Data":"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"}
Jul 27 20:55:12.648967 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.648876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:12.648967 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.648907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" event={"ID":"71655fca-159d-488a-b111-ba3338211a09","Type":"ContainerStarted","Data":"d9ea27f27dfd592cc01c32b0edbedbc15bc6c860708c3d13f4c008ded2eb3278"}
Jul 27 20:55:12.650318 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.650293    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerID="090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c" exitCode=0
Jul 27 20:55:12.650416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.650337    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"
Jul 27 20:55:12.650416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.650366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerDied","Data":"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"}
Jul 27 20:55:12.650416 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.650407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc" event={"ID":"e6e2df55-9e86-4b3e-b821-4cc7c4f71fff","Type":"ContainerDied","Data":"905608daff9deaaccc6dde7b575f56b1fc98f798389aa9f394e7378e06781d63"}
Jul 27 20:55:12.650517 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.650437    2560 scope.go:117] "RemoveContainer" containerID="090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"
Jul 27 20:55:12.660179 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.660160    2560 scope.go:117] "RemoveContainer" containerID="330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"
Jul 27 20:55:12.668866 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.668793    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" podStartSLOduration=2.107310014 podStartE2EDuration="2.668774421s" totalImagesPullingTime="561.464407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:12.666441326 +0000 UTC m=+2245.516208160" watchObservedRunningTime="2026-07-27 20:55:12.668774421 +0000 UTC m=+2245.518541254"
Jul 27 20:55:12.670496 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.670468    2560 scope.go:117] "RemoveContainer" containerID="090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"
Jul 27 20:55:12.671079 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:55:12.671004    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c\": container with ID starting with 090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c not found: ID does not exist" containerID="090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"
Jul 27 20:55:12.671079 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.671041    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c"} err="failed to get container status \"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c\": rpc error: code = NotFound desc = could not find container \"090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c\": container with ID starting with 090cfa75e27ed277699a18ae8ba40636cd73749f3713460c289110573526801c not found: ID does not exist"
Jul 27 20:55:12.671079 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.671066    2560 scope.go:117] "RemoveContainer" containerID="330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"
Jul 27 20:55:12.671379 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:55:12.671354    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647\": container with ID starting with 330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647 not found: ID does not exist" containerID="330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"
Jul 27 20:55:12.671429 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.671391    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647"} err="failed to get container status \"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647\": rpc error: code = NotFound desc = could not find container \"330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647\": container with ID starting with 330103aff4b418c902e849500c6afb85e2dac4d3fd558aaa1bca7e6dda98e647 not found: ID does not exist"
Jul 27 20:55:12.682180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.682143    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:55:12.688862 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:12.688833    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-704ca2-openshift-default-99d69789d-tfbqc"]
Jul 27 20:55:13.726767 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:13.726726    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" path="/var/lib/kubelet/pods/e6e2df55-9e86-4b3e-b821-4cc7c4f71fff/volumes"
Jul 27 20:55:15.831993 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.831952    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:55:15.832425 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.832409    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:15.832465 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.832426    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:15.832520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.832497    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:15.832520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.832506    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:15.836983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.836966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.839487 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.839461    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d4c2c3-openshift-default-dockercfg-9nztk\""
Jul 27 20:55:15.845882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.845859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:55:15.949112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfr2\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949328 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:15.949459 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:15.949415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.050959 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.050914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfr2\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.050959 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.050962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.050984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051625 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051625 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051697 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051834 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.051891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.051851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.053393 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.053365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.053729 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.053704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.059122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.059078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.062827 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.059313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfr2\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2\") pod \"e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") " pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.149444 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.149401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:16.335113 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.335084    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:55:16.336936 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:55:16.336910    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf410f7c_17fc_4839_99dd_72ed23aab721.slice/crio-980c5b5cdf777ffd80898e3f71a948c2723aa4ecbf8cbe87a074747be3d86042 WatchSource:0}: Error finding container 980c5b5cdf777ffd80898e3f71a948c2723aa4ecbf8cbe87a074747be3d86042: Status 404 returned error can't find the container with id 980c5b5cdf777ffd80898e3f71a948c2723aa4ecbf8cbe87a074747be3d86042
Jul 27 20:55:16.339478 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.339435    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:16.339550 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.339535    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:16.339597 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.339575    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:16.681413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.680320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerStarted","Data":"e44c5ed92abe8f30adc607ea100667221c87651878b3bf79b37d8008a7b263ce"}
Jul 27 20:55:16.681413 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.680391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerStarted","Data":"980c5b5cdf777ffd80898e3f71a948c2723aa4ecbf8cbe87a074747be3d86042"}
Jul 27 20:55:16.702228 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:16.702165    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podStartSLOduration=1.702147132 podStartE2EDuration="1.702147132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:16.701374606 +0000 UTC m=+2249.551141438" watchObservedRunningTime="2026-07-27 20:55:16.702147132 +0000 UTC m=+2249.551913963"
Jul 27 20:55:17.150195 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:17.150159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:17.151983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:17.151939    2560 patch_prober.go:28] interesting pod/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:55:17.152147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:17.152023    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 27 20:55:18.154838 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.154789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:18.666490 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.666450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:18.695306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.695274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:18.696700 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.696674    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:18.755520 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.755490    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:55:18.755945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.755927    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:18.755945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.755944    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e2df55-9e86-4b3e-b821-4cc7c4f71fff" containerName="istio-proxy"
Jul 27 20:55:18.759254 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.759230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.762475 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.762452    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d4c2c3-serving-cert\""
Jul 27 20:55:18.769116 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.769089    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:55:18.888570 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.888526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.888760 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.888636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.888977 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.888941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4jn\" (UniqueName: \"kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.990360 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.990261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.990360 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.990305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.990588 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.990380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4jn\" (UniqueName: \"kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.990945 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.990918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:18.992788 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:18.992765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:19.004225 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:19.004202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4jn\" (UniqueName: \"kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn\") pod \"maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:19.077016 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:19.076983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:19.222990 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:19.222771    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:55:19.225983 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:55:19.225938    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6437ceeb_cf7a_4314_b8e7_c1c1f9c18d33.slice/crio-204f8f6fd4b1a365f60fa583fa182abfd25f8e7adf5cf9075f670de8b9896063 WatchSource:0}: Error finding container 204f8f6fd4b1a365f60fa583fa182abfd25f8e7adf5cf9075f670de8b9896063: Status 404 returned error can't find the container with id 204f8f6fd4b1a365f60fa583fa182abfd25f8e7adf5cf9075f670de8b9896063
Jul 27 20:55:19.702685 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:19.702655    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" event={"ID":"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33","Type":"ContainerStarted","Data":"204f8f6fd4b1a365f60fa583fa182abfd25f8e7adf5cf9075f670de8b9896063"}
Jul 27 20:55:20.709223 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:20.709184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" event={"ID":"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33","Type":"ContainerStarted","Data":"d1111c193fede395c547b7eb767e3214bcf7f745fef4b0baa871d7b634f4da46"}
Jul 27 20:55:20.730175 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:20.730127    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" podStartSLOduration=2.274158418 podStartE2EDuration="2.730112293s" totalImagesPullingTime="455.953875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:20.728199372 +0000 UTC m=+2253.577966206" watchObservedRunningTime="2026-07-27 20:55:20.730112293 +0000 UTC m=+2253.579879122"
Jul 27 20:55:21.716126 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.716092    2560 generic.go:356] "Generic (PLEG): container finished" podID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerID="ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82" exitCode=0
Jul 27 20:55:21.716591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.716162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerDied","Data":"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"}
Jul 27 20:55:21.716591 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.716356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:21.716774 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.716757    2560 scope.go:117] "RemoveContainer" containerID="ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"
Jul 27 20:55:21.717914 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.717885    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:21.718043 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.717954    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:21.718043 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:21.717987    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:22.723153 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:22.723109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerStarted","Data":"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"}
Jul 27 20:55:22.786207 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:22.786164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:22.787273 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:22.787242    2560 patch_prober.go:28] interesting pod/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:55:22.787394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:22.787302    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 27 20:55:23.785480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:23.785426    2560 patch_prober.go:28] interesting pod/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:55:23.786078 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:23.785509    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 27 20:55:24.785961 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:24.785917    2560 patch_prober.go:28] interesting pod/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:55:24.786484 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:24.785994    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 27 20:55:25.638959 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:25.638905    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:25.785932 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:25.785880    2560 patch_prober.go:28] interesting pod/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:55:25.786653 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:25.786220    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 27 20:55:26.790013 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:26.789980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:26.791023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:26.791000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:55:27.728882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:27.728854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:55:32.776829 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.776773    2560 generic.go:356] "Generic (PLEG): container finished" podID="df410f7c-17fc-4839-99dd-72ed23aab721" containerID="e44c5ed92abe8f30adc607ea100667221c87651878b3bf79b37d8008a7b263ce" exitCode=0
Jul 27 20:55:32.777245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.776854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerDied","Data":"e44c5ed92abe8f30adc607ea100667221c87651878b3bf79b37d8008a7b263ce"}
Jul 27 20:55:32.777288 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.777254    2560 scope.go:117] "RemoveContainer" containerID="e44c5ed92abe8f30adc607ea100667221c87651878b3bf79b37d8008a7b263ce"
Jul 27 20:55:32.778309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.778282    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:32.778367 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.778356    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:32.778427 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:32.778415    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:55:33.695985 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:33.695949    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:33.787891 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:33.787850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerStarted","Data":"687837d89360c16de979255553c4506ba9913d6a94dad951362fc2dcef0d9d37"}
Jul 27 20:55:34.149907 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:34.149866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:34.151119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:34.151089    2560 patch_prober.go:28] interesting pod/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:55:34.151272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:34.151149    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 27 20:55:35.149986 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:35.149950    2560 patch_prober.go:28] interesting pod/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:55:35.150404 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:35.150012    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 27 20:55:36.150236 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:36.150197    2560 patch_prober.go:28] interesting pod/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:55:36.150641 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:36.150259    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 27 20:55:37.150703 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:37.150665    2560 patch_prober.go:28] interesting pod/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:55:37.151119 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:37.150728    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 27 20:55:38.153872 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:38.153831    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:38.154270 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:38.154179    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:38.154835 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:38.154798    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:55:46.871090 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:46.871043    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:46.871565 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:46.871255    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" podUID="71655fca-159d-488a-b111-ba3338211a09" containerName="maas-api" containerID="cri-o://4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe" gracePeriod=30
Jul 27 20:55:47.125887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.125747    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:47.170733 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.170700    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") pod \"71655fca-159d-488a-b111-ba3338211a09\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") "
Jul 27 20:55:47.170913 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.170892    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca\") pod \"71655fca-159d-488a-b111-ba3338211a09\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") "
Jul 27 20:55:47.170959 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.170925    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plkg2\" (UniqueName: \"kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2\") pod \"71655fca-159d-488a-b111-ba3338211a09\" (UID: \"71655fca-159d-488a-b111-ba3338211a09\") "
Jul 27 20:55:47.171317 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.171286    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71655fca-159d-488a-b111-ba3338211a09" (UID: "71655fca-159d-488a-b111-ba3338211a09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:47.172836 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.172795    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71655fca-159d-488a-b111-ba3338211a09" (UID: "71655fca-159d-488a-b111-ba3338211a09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:55:47.172969 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.172943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2" (OuterVolumeSpecName: "kube-api-access-plkg2") pod "71655fca-159d-488a-b111-ba3338211a09" (UID: "71655fca-159d-488a-b111-ba3338211a09"). InnerVolumeSpecName "kube-api-access-plkg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:47.272581 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.272539    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71655fca-159d-488a-b111-ba3338211a09-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:47.272581 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.272574    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71655fca-159d-488a-b111-ba3338211a09-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:47.272581 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.272589    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plkg2\" (UniqueName: \"kubernetes.io/projected/71655fca-159d-488a-b111-ba3338211a09-kube-api-access-plkg2\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:55:47.859570 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.859537    2560 generic.go:356] "Generic (PLEG): container finished" podID="71655fca-159d-488a-b111-ba3338211a09" containerID="4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe" exitCode=0
Jul 27 20:55:47.859779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.859621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"
Jul 27 20:55:47.859779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.859620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" event={"ID":"71655fca-159d-488a-b111-ba3338211a09","Type":"ContainerDied","Data":"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"}
Jul 27 20:55:47.859779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.859686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt" event={"ID":"71655fca-159d-488a-b111-ba3338211a09","Type":"ContainerDied","Data":"d9ea27f27dfd592cc01c32b0edbedbc15bc6c860708c3d13f4c008ded2eb3278"}
Jul 27 20:55:47.859779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.859714    2560 scope.go:117] "RemoveContainer" containerID="4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"
Jul 27 20:55:47.868978 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.868957    2560 scope.go:117] "RemoveContainer" containerID="4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"
Jul 27 20:55:47.869288 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:55:47.869267    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe\": container with ID starting with 4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe not found: ID does not exist" containerID="4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"
Jul 27 20:55:47.869368 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.869299    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe"} err="failed to get container status \"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe\": rpc error: code = NotFound desc = could not find container \"4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe\": container with ID starting with 4940aa6ba5d10d41c2f420e91443d46a662bd38008688e9f11615c8c197a82fe not found: ID does not exist"
Jul 27 20:55:47.876288 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.876266    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:47.878003 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.877983    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc8abd-687b4f8dcc-5lttt"]
Jul 27 20:55:47.935499 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:47.935470    2560 scope.go:117] "RemoveContainer" containerID="6a2481c80c94764c317c7682ebc39f09809c99ba43b4d1a0b6b4f0f1eba436be"
Jul 27 20:55:49.727043 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:49.727011    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71655fca-159d-488a-b111-ba3338211a09" path="/var/lib/kubelet/pods/71655fca-159d-488a-b111-ba3338211a09/volumes"
Jul 27 20:55:57.974369 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:57.974332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:55:57.974950 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:55:57.974581    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy" containerID="cri-o://8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1" gracePeriod=30
Jul 27 20:56:03.228905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.228880    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:56:03.320826 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320726    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.320826 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320771    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320889    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4crp\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321040 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.320997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo\") pod \"cd38041e-b64f-4ab6-a69d-7cec66a44889\" (UID: \"cd38041e-b64f-4ab6-a69d-7cec66a44889\") "
Jul 27 20:56:03.321338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:03.321338 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321173    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:03.321551 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321376    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:03.321551 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321458    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:56:03.321551 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321520    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.321551 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321542    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.321762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.321555    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.322359 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.322331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:03.323073 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.323046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:56:03.323417 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.323400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:56:03.323539 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.323516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:03.323596 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.323520    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp" (OuterVolumeSpecName: "kube-api-access-t4crp") pod "cd38041e-b64f-4ab6-a69d-7cec66a44889" (UID: "cd38041e-b64f-4ab6-a69d-7cec66a44889"). InnerVolumeSpecName "kube-api-access-t4crp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:56:03.422355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422322    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd38041e-b64f-4ab6-a69d-7cec66a44889-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.422355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422350    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.422355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422358    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.422355 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422366    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.422636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422374    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd38041e-b64f-4ab6-a69d-7cec66a44889-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.422636 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.422382    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4crp\" (UniqueName: \"kubernetes.io/projected/cd38041e-b64f-4ab6-a69d-7cec66a44889-kube-api-access-t4crp\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:03.941871 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.941828    2560 generic.go:356] "Generic (PLEG): container finished" podID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerID="8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1" exitCode=0
Jul 27 20:56:03.941871 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.941876    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"
Jul 27 20:56:03.942122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.941888    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerDied","Data":"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"}
Jul 27 20:56:03.942122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.941937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9" event={"ID":"cd38041e-b64f-4ab6-a69d-7cec66a44889","Type":"ContainerDied","Data":"3d4b7478063351cf2d380c85dcf20c017057b8c24dda47942fd57d9db1fa910b"}
Jul 27 20:56:03.942122 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.941964    2560 scope.go:117] "RemoveContainer" containerID="8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"
Jul 27 20:56:03.950554 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.950536    2560 scope.go:117] "RemoveContainer" containerID="ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"
Jul 27 20:56:03.958372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.958354    2560 scope.go:117] "RemoveContainer" containerID="8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"
Jul 27 20:56:03.958630 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:56:03.958604    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1\": container with ID starting with 8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1 not found: ID does not exist" containerID="8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"
Jul 27 20:56:03.958734 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.958633    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1"} err="failed to get container status \"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1\": rpc error: code = NotFound desc = could not find container \"8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1\": container with ID starting with 8ee6f78e0238a5ddf7e1a69d2709f5ac898995f1ac82dc13e9de2ca854689ce1 not found: ID does not exist"
Jul 27 20:56:03.958734 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.958651    2560 scope.go:117] "RemoveContainer" containerID="ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"
Jul 27 20:56:03.958902 ip-10-0-143-74 kubenswrapper[2560]: E0727 20:56:03.958886    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82\": container with ID starting with ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82 not found: ID does not exist" containerID="ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"
Jul 27 20:56:03.958961 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.958907    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82"} err="failed to get container status \"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82\": rpc error: code = NotFound desc = could not find container \"ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82\": container with ID starting with ffbc7a59c6f73a3f4b14cc0d489d006fcb41b9b02319243421a37336ebf0bf82 not found: ID does not exist"
Jul 27 20:56:03.964600 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.964574    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:56:03.969983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:03.969956    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cc8abd-openshift-default-dbc957b86-smff9"]
Jul 27 20:56:05.727485 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:05.727448    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" path="/var/lib/kubelet/pods/cd38041e-b64f-4ab6-a69d-7cec66a44889/volumes"
Jul 27 20:56:13.799612 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:13.799568    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:56:13.800147 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:13.799834    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" podUID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" containerName="maas-api" containerID="cri-o://d1111c193fede395c547b7eb767e3214bcf7f745fef4b0baa871d7b634f4da46" gracePeriod=30
Jul 27 20:56:13.993973 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:13.993943    2560 generic.go:356] "Generic (PLEG): container finished" podID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" containerID="d1111c193fede395c547b7eb767e3214bcf7f745fef4b0baa871d7b634f4da46" exitCode=0
Jul 27 20:56:13.994127 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:13.994010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" event={"ID":"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33","Type":"ContainerDied","Data":"d1111c193fede395c547b7eb767e3214bcf7f745fef4b0baa871d7b634f4da46"}
Jul 27 20:56:14.058909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.058842    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:56:14.125774 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.125743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca\") pod \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") "
Jul 27 20:56:14.125983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.125886    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx4jn\" (UniqueName: \"kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn\") pod \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") "
Jul 27 20:56:14.125983 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.125926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls\") pod \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\" (UID: \"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33\") "
Jul 27 20:56:14.126121 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.126074    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" (UID: "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:56:14.126208 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.126190    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:14.128052 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.128018    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" (UID: "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:56:14.128180 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.128111    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn" (OuterVolumeSpecName: "kube-api-access-dx4jn") pod "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" (UID: "6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33"). InnerVolumeSpecName "kube-api-access-dx4jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:56:14.226910 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.226872    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:14.226910 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:14.226905    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4jn\" (UniqueName: \"kubernetes.io/projected/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33-kube-api-access-dx4jn\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:15.003897 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.003864    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"
Jul 27 20:56:15.003897 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.003873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b" event={"ID":"6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33","Type":"ContainerDied","Data":"204f8f6fd4b1a365f60fa583fa182abfd25f8e7adf5cf9075f670de8b9896063"}
Jul 27 20:56:15.004430 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.003953    2560 scope.go:117] "RemoveContainer" containerID="d1111c193fede395c547b7eb767e3214bcf7f745fef4b0baa871d7b634f4da46"
Jul 27 20:56:15.026481 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.026452    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:56:15.030291 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.030247    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4c2c3-886cd788b-htx4b"]
Jul 27 20:56:15.726425 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:15.726393    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" path="/var/lib/kubelet/pods/6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33/volumes"
Jul 27 20:56:24.931660 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:24.931625    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:56:24.932084 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:24.931949    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy" containerID="cri-o://687837d89360c16de979255553c4506ba9913d6a94dad951362fc2dcef0d9d37" gracePeriod=30
Jul 27 20:56:25.815343 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815310    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 20:56:25.815706 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815693    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815708    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815720    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815725    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815733    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71655fca-159d-488a-b111-ba3338211a09" containerName="maas-api"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815739    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71655fca-159d-488a-b111-ba3338211a09" containerName="maas-api"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815750    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" containerName="maas-api"
Jul 27 20:56:25.815779 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815755    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" containerName="maas-api"
Jul 27 20:56:25.816062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815835    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6437ceeb-cf7a-4314-b8e7-c1c1f9c18d33" containerName="maas-api"
Jul 27 20:56:25.816062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815843    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.816062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815851    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd38041e-b64f-4ab6-a69d-7cec66a44889" containerName="istio-proxy"
Jul 27 20:56:25.816062 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.815857    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="71655fca-159d-488a-b111-ba3338211a09" containerName="maas-api"
Jul 27 20:56:25.819044 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.819015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.821786 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.821763    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-132aad-openshift-default-dockercfg-vb8sq\""
Jul 27 20:56:25.830703 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.830675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 20:56:25.939392 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzldg\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939466    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:25.939927 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:25.939803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.040865 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.040865 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzldg\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.040969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041443 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041554 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.041906 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.041882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.043266 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.043236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.043558 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.043540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.049556 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.049535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzldg\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.049762 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.049745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token\") pod \"e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") " pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.129994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.129952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:26.285233 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.285205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 20:56:26.287323 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:56:26.287296    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdc1096_182c_4115_99a1_e49df98662c3.slice/crio-7e3e5dd23e3a045b70730fb0ea8051640cc77f2782f77365eb179e0681243729 WatchSource:0}: Error finding container 7e3e5dd23e3a045b70730fb0ea8051640cc77f2782f77365eb179e0681243729: Status 404 returned error can't find the container with id 7e3e5dd23e3a045b70730fb0ea8051640cc77f2782f77365eb179e0681243729
Jul 27 20:56:26.289694 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.289646    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:26.289791 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.289744    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:26.289854 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:26.289831    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:27.062294 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:27.062259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerStarted","Data":"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"}
Jul 27 20:56:27.062659 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:27.062320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerStarted","Data":"7e3e5dd23e3a045b70730fb0ea8051640cc77f2782f77365eb179e0681243729"}
Jul 27 20:56:27.084053 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:27.083996    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podStartSLOduration=2.08397628 podStartE2EDuration="2.08397628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:27.083107166 +0000 UTC m=+2319.932873989" watchObservedRunningTime="2026-07-27 20:56:27.08397628 +0000 UTC m=+2319.933743111"
Jul 27 20:56:27.130471 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:27.130435    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:27.135490 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:27.135461    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:28.067474 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:28.067439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:28.068432 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:28.068413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:29.229231 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.229196    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 20:56:29.232561 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.232540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.235228 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.235201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-132aad-serving-cert\""
Jul 27 20:56:29.235348 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.235209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:56:29.236245 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.236228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q9t2z\""
Jul 27 20:56:29.236346 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.236245    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:56:29.244112 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.244078    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 20:56:29.372277 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.372246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.372477 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.372405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.372540 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.372499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjff\" (UniqueName: \"kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.473691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.473648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.473934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.473747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.473934 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.473793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjff\" (UniqueName: \"kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.474335 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.474311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.476287 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.476263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.486691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.486638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjff\" (UniqueName: \"kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff\") pod \"maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.545308 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.545271    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:29.695306 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:29.695276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 20:56:29.697237 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:56:29.697209    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1682669_bd42_4fc3_9faa_5f9043567482.slice/crio-961228911a43b7c5e53c95de142d579087ee14f36dbfa8143fb3d0cd21f6ef95 WatchSource:0}: Error finding container 961228911a43b7c5e53c95de142d579087ee14f36dbfa8143fb3d0cd21f6ef95: Status 404 returned error can't find the container with id 961228911a43b7c5e53c95de142d579087ee14f36dbfa8143fb3d0cd21f6ef95
Jul 27 20:56:30.079972 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.079940    2560 generic.go:356] "Generic (PLEG): container finished" podID="df410f7c-17fc-4839-99dd-72ed23aab721" containerID="687837d89360c16de979255553c4506ba9913d6a94dad951362fc2dcef0d9d37" exitCode=0
Jul 27 20:56:30.080244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.080016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerDied","Data":"687837d89360c16de979255553c4506ba9913d6a94dad951362fc2dcef0d9d37"}
Jul 27 20:56:30.080244 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.080088    2560 scope.go:117] "RemoveContainer" containerID="e44c5ed92abe8f30adc607ea100667221c87651878b3bf79b37d8008a7b263ce"
Jul 27 20:56:30.081946 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.081907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" event={"ID":"a1682669-bd42-4fc3-9faa-5f9043567482","Type":"ContainerStarted","Data":"961228911a43b7c5e53c95de142d579087ee14f36dbfa8143fb3d0cd21f6ef95"}
Jul 27 20:56:30.199586 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.199561    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:56:30.280604 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280566    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsfr2\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280629    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280652    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280700    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281075 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.280728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281115    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert\") pod \"df410f7c-17fc-4839-99dd-72ed23aab721\" (UID: \"df410f7c-17fc-4839-99dd-72ed23aab721\") "
Jul 27 20:56:30.281372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:30.281705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281557    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.281705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281579    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.281705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281593    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.281705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281608    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data" (OuterVolumeSpecName: "istio-data") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:30.281705 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.281624    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:56:30.282944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.282922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2" (OuterVolumeSpecName: "kube-api-access-lsfr2") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "kube-api-access-lsfr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:56:30.283252 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.283234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:56:30.283371 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.283343    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token" (OuterVolumeSpecName: "istio-token") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:56:30.283480 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.283376    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df410f7c-17fc-4839-99dd-72ed23aab721" (UID: "df410f7c-17fc-4839-99dd-72ed23aab721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:56:30.382920 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382884    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.382920 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382915    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df410f7c-17fc-4839-99dd-72ed23aab721-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.382920 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382926    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfr2\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-kube-api-access-lsfr2\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.383146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382939    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df410f7c-17fc-4839-99dd-72ed23aab721-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.383146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382948    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df410f7c-17fc-4839-99dd-72ed23aab721-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:30.383146 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:30.382956    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df410f7c-17fc-4839-99dd-72ed23aab721-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 20:56:31.087994 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.087954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68" event={"ID":"df410f7c-17fc-4839-99dd-72ed23aab721","Type":"ContainerDied","Data":"980c5b5cdf777ffd80898e3f71a948c2723aa4ecbf8cbe87a074747be3d86042"}
Jul 27 20:56:31.088196 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.088037    2560 scope.go:117] "RemoveContainer" containerID="687837d89360c16de979255553c4506ba9913d6a94dad951362fc2dcef0d9d37"
Jul 27 20:56:31.088196 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.087968    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"
Jul 27 20:56:31.089747 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.089718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" event={"ID":"a1682669-bd42-4fc3-9faa-5f9043567482","Type":"ContainerStarted","Data":"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"}
Jul 27 20:56:31.089909 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.089781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:31.110439 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.110396    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" podStartSLOduration=1.676935965 podStartE2EDuration="2.110380998s" totalImagesPullingTime="433.445033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:31.109070342 +0000 UTC m=+2323.958837174" watchObservedRunningTime="2026-07-27 20:56:31.110380998 +0000 UTC m=+2323.960147828"
Jul 27 20:56:31.124989 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.124961    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:56:31.128408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.128381    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4c2c3-openshift-default-6b7995855b-fhm68"]
Jul 27 20:56:31.726448 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:31.726404    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" path="/var/lib/kubelet/pods/df410f7c-17fc-4839-99dd-72ed23aab721/volumes"
Jul 27 20:56:34.095966 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.095922    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 20:56:34.096533 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096512    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.096590 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096538    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.096590 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096576    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.096590 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096585    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.096707 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096693    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.096745 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.096711    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df410f7c-17fc-4839-99dd-72ed23aab721" containerName="istio-proxy"
Jul 27 20:56:34.117137 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.117110    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 20:56:34.117298 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.117204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.119918 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.119896    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-66b603-openshift-default-dockercfg-b7jv9\""
Jul 27 20:56:34.216133 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216315 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216315 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216315 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216315 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216315 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jpm\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.216492 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.216387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.317833 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.317833 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jpm\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.317992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318094 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318410 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318410 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318410 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318410 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.318604 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.318585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.320309 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.320292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.320682 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.320667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.325845 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.325820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.325845 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.325842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jpm\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm\") pod \"e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") " pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.427821 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.427784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:34.609801 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.609778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 20:56:34.612061 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:56:34.612030    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f8730d_3dd9_428f_8638_9204c7cc29f1.slice/crio-e088988d2031e92103dce0342939008f1817f3aa0d23afd07957326b88f49914 WatchSource:0}: Error finding container e088988d2031e92103dce0342939008f1817f3aa0d23afd07957326b88f49914: Status 404 returned error can't find the container with id e088988d2031e92103dce0342939008f1817f3aa0d23afd07957326b88f49914
Jul 27 20:56:34.614899 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.614859    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:34.614981 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.614931    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:34.614981 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:34.614959    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:35.113708 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:35.113608    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerStarted","Data":"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"}
Jul 27 20:56:35.113708 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:35.113675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerStarted","Data":"e088988d2031e92103dce0342939008f1817f3aa0d23afd07957326b88f49914"}
Jul 27 20:56:35.133166 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:35.133113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podStartSLOduration=1.133095614 podStartE2EDuration="1.133095614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:35.132400904 +0000 UTC m=+2327.982167776" watchObservedRunningTime="2026-07-27 20:56:35.133095614 +0000 UTC m=+2327.982862444"
Jul 27 20:56:35.428276 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:35.428231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:35.432906 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:35.432882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:36.119045 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:36.119014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:36.120157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:36.120139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:37.102185 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.102159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 20:56:37.109214 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.109183    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 20:56:37.111907 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.111882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.114256 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.114233    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-66b603-serving-cert\""
Jul 27 20:56:37.123372 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.123340    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 20:56:37.145804 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.145773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.146002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.145913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.146002 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.145963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pnn\" (UniqueName: \"kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.247622 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.247536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.247887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.247667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pnn\" (UniqueName: \"kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.247887 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.247739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.248561 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.248535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.251085 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.251058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.258729 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.258667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pnn\" (UniqueName: \"kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn\") pod \"maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.427856 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.427795    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:37.592680 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:37.592654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 20:56:37.595040 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:56:37.595010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3299c8d5_27d2_4645_900a_837a2f2e42b5.slice/crio-59ca2931789a512c952ac30b5a8bf34b4cde80d690bc9ce8d44c78c6cecfe252 WatchSource:0}: Error finding container 59ca2931789a512c952ac30b5a8bf34b4cde80d690bc9ce8d44c78c6cecfe252: Status 404 returned error can't find the container with id 59ca2931789a512c952ac30b5a8bf34b4cde80d690bc9ce8d44c78c6cecfe252
Jul 27 20:56:38.133606 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:38.133564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" event={"ID":"3299c8d5-27d2-4645-900a-837a2f2e42b5","Type":"ContainerStarted","Data":"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"}
Jul 27 20:56:38.134044 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:38.133634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" event={"ID":"3299c8d5-27d2-4645-900a-837a2f2e42b5","Type":"ContainerStarted","Data":"59ca2931789a512c952ac30b5a8bf34b4cde80d690bc9ce8d44c78c6cecfe252"}
Jul 27 20:56:38.134044 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:38.133761    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:38.152479 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:38.152430    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" podStartSLOduration=0.803709484 podStartE2EDuration="1.152413357s" totalImagesPullingTime="348.703873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:38.151087795 +0000 UTC m=+2331.000854627" watchObservedRunningTime="2026-07-27 20:56:38.152413357 +0000 UTC m=+2331.002180187"
Jul 27 20:56:44.108998 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:44.108951    2560 patch_prober.go:28] interesting pod/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:56:44.109398 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:44.109013    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:56:44.144618 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:44.144586    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 20:56:46.180501 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.180461    2560 generic.go:356] "Generic (PLEG): container finished" podID="4cdc1096-182c-4115-99a1-e49df98662c3" containerID="fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729" exitCode=0
Jul 27 20:56:46.180942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.180504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerDied","Data":"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"}
Jul 27 20:56:46.180942 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.180925    2560 scope.go:117] "RemoveContainer" containerID="fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"
Jul 27 20:56:46.181970 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.181942    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:46.182023 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.182008    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:46.182058 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:46.182040    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:47.190548 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:47.190516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerStarted","Data":"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"}
Jul 27 20:56:47.971825 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:47.971765    2560 scope.go:117] "RemoveContainer" containerID="230b8f231d30ab29785f3133e97269cfe63fe5a2fd58661d10477451a5cf4443"
Jul 27 20:56:48.130431 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:48.130402    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:48.130922 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:48.130897    2560 patch_prober.go:28] interesting pod/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 27 20:56:48.131012 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:48.130957    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 27 20:56:49.130737 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:49.130695    2560 patch_prober.go:28] interesting pod/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 27 20:56:49.131157 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:49.130752    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 27 20:56:50.130661 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:50.130626    2560 patch_prober.go:28] interesting pod/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 27 20:56:50.130944 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:50.130920    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 27 20:56:51.131066 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:51.131028    2560 patch_prober.go:28] interesting pod/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 27 20:56:51.131452 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:51.131086    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 27 20:56:52.134882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:52.134851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:52.135267 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:52.135129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:52.136008 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:52.135990    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 20:56:52.160964 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:52.160920    2560 patch_prober.go:28] interesting pod/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:56:52.161171 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:52.160974    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:56:54.238905 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.238869    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerID="a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353" exitCode=0
Jul 27 20:56:54.239445 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.238938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerDied","Data":"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"}
Jul 27 20:56:54.239445 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.239326    2560 scope.go:117] "RemoveContainer" containerID="a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"
Jul 27 20:56:54.240346 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.240312    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:54.240470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.240403    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:54.240470 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:54.240437    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 20:56:55.246234 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:55.246201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerStarted","Data":"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"}
Jul 27 20:56:55.428507 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:55.428467    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:55.428731 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:55.428708    2560 patch_prober.go:28] interesting pod/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 27 20:56:55.428780 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:55.428759    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 27 20:56:56.428726 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:56.428681    2560 patch_prober.go:28] interesting pod/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 27 20:56:56.429272 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:56.428753    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 27 20:56:57.428949 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:57.428905    2560 patch_prober.go:28] interesting pod/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 27 20:56:57.429369 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:57.428966    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 27 20:56:58.428641 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:58.428603    2560 patch_prober.go:28] interesting pod/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 27 20:56:58.428882 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:58.428672    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 27 20:56:59.432452 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:59.432425    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:59.432971 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:59.432838    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:56:59.433421 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:56:59.433400    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 20:57:10.607381 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.607348    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 20:57:10.617408 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.617384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.620457 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.620426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"kube-root-ca.crt\""
Jul 27 20:57:10.620631 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.620439    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"openshift-service-ca.crt\""
Jul 27 20:57:10.621691 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.621669    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"default-dockercfg-rvfc5\""
Jul 27 20:57:10.621859 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.621800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"auth-test-model-132aad-kserve-self-signed-certs\""
Jul 27 20:57:10.622296 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.622270    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 20:57:10.702213 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.702394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.702394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.702394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.702394 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrgt\" (UniqueName: \"kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.702539 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.702400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803635 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrgt\" (UniqueName: \"kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.803852 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.803754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.804154 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.804134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.804202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.804149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.804202 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.804193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.805975 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.805952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.806308 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.806292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.811555 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.811526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrgt\" (UniqueName: \"kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt\") pod \"auth-test-model-132aad-kserve-5946cd67bc-lv7bb\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") " pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:10.929030 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:10.928991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:11.064027 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:11.063996    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 20:57:11.066137 ip-10-0-143-74 kubenswrapper[2560]: W0727 20:57:11.066101    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf9a02ed_b575_4b29_9608_c7e5a219149e.slice/crio-059dede44c59640a0b32b9c4c15c7d2600f4ea9f88c282ee39b7384ca989f7f7 WatchSource:0}: Error finding container 059dede44c59640a0b32b9c4c15c7d2600f4ea9f88c282ee39b7384ca989f7f7: Status 404 returned error can't find the container with id 059dede44c59640a0b32b9c4c15c7d2600f4ea9f88c282ee39b7384ca989f7f7
Jul 27 20:57:11.335661 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:11.335564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" event={"ID":"af9a02ed-b575-4b29-9608-c7e5a219149e","Type":"ContainerStarted","Data":"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"}
Jul 27 20:57:11.335661 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:11.335610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" event={"ID":"af9a02ed-b575-4b29-9608-c7e5a219149e","Type":"ContainerStarted","Data":"059dede44c59640a0b32b9c4c15c7d2600f4ea9f88c282ee39b7384ca989f7f7"}
Jul 27 20:57:11.353224 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:11.352902    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" podStartSLOduration=1.352880826 podStartE2EDuration="1.352880826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:11.3526949 +0000 UTC m=+2364.202461731" watchObservedRunningTime="2026-07-27 20:57:11.352880826 +0000 UTC m=+2364.202647658"
Jul 27 20:57:20.929574 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:20.929532    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:20.930113 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:20.929591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:20.942209 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:20.942182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 20:57:21.397830 ip-10-0-143-74 kubenswrapper[2560]: I0727 20:57:21.397783    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 21:00:00.133707 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.133627    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"]
Jul 27 21:00:00.137706 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.137685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"
Jul 27 21:00:00.140213 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.140196    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xw8dl\""
Jul 27 21:00:00.146857 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.146838    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"
Jul 27 21:00:00.156943 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.156918    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"]
Jul 27 21:00:00.282983 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:00.282783    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"]
Jul 27 21:00:00.285238 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:00:00.285206    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8756535_e34c_4b60_ad5f_077fb1fd99cf.slice/crio-95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63 WatchSource:0}: Error finding container 95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63: Status 404 returned error can't find the container with id 95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63
Jul 27 21:00:01.181756 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:01.181724    2560 generic.go:356] "Generic (PLEG): container finished" podID="f8756535-e34c-4b60-ad5f-077fb1fd99cf" containerID="a7c77aa288c436dc1b30b6b52d6c70a0690f011afcc1f09b130f69d17ad5723f" exitCode=0
Jul 27 21:00:01.182077 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:01.181772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s" event={"ID":"f8756535-e34c-4b60-ad5f-077fb1fd99cf","Type":"ContainerDied","Data":"a7c77aa288c436dc1b30b6b52d6c70a0690f011afcc1f09b130f69d17ad5723f"}
Jul 27 21:00:01.182077 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:01.181833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s" event={"ID":"f8756535-e34c-4b60-ad5f-077fb1fd99cf","Type":"ContainerStarted","Data":"95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63"}
Jul 27 21:00:02.314724 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:02.314697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"
Jul 27 21:00:03.192457 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:03.192432    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s"
Jul 27 21:00:03.192669 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:03.192473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-cz67s" event={"ID":"f8756535-e34c-4b60-ad5f-077fb1fd99cf","Type":"ContainerDied","Data":"95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63"}
Jul 27 21:00:03.192669 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:03.192523    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d8938cb61bc46d3191a8d10c996d2578b4fbbb572a85e8713d7aea3be62d63"
Jul 27 21:00:48.047881 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:00:48.047848    2560 scope.go:117] "RemoveContainer" containerID="56748ade7f0abb6ca8888ff48d008b431fe414fab435cc3a08d405397f19d270"
Jul 27 21:02:08.404780 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:08.404747    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 21:02:23.425432 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.425366    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" podUID="af9a02ed-b575-4b29-9608-c7e5a219149e" containerName="main" containerID="cri-o://baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8" gracePeriod=45
Jul 27 21:02:23.674149 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.674125    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 21:02:23.741135 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741135 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741317 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741317 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lrgt\" (UniqueName: \"kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741317 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741237    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741317 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home\") pod \"af9a02ed-b575-4b29-9608-c7e5a219149e\" (UID: \"af9a02ed-b575-4b29-9608-c7e5a219149e\") "
Jul 27 21:02:23.741542 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741440    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:02:23.741542 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache" (OuterVolumeSpecName: "model-cache") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:02:23.741659 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home" (OuterVolumeSpecName: "home") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:02:23.741659 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741623    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.741659 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.741637    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.743321 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.743302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt" (OuterVolumeSpecName: "kube-api-access-9lrgt") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "kube-api-access-9lrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:02:23.743669 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.743654    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:02:23.743741 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.743662    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm" (OuterVolumeSpecName: "dshm") pod "af9a02ed-b575-4b29-9608-c7e5a219149e" (UID: "af9a02ed-b575-4b29-9608-c7e5a219149e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:02:23.848304 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.842411    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9a02ed-b575-4b29-9608-c7e5a219149e-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.848304 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.842482    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.848304 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.842498    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrgt\" (UniqueName: \"kubernetes.io/projected/af9a02ed-b575-4b29-9608-c7e5a219149e-kube-api-access-9lrgt\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.848304 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.842512    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9a02ed-b575-4b29-9608-c7e5a219149e-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:02:23.907115 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.907080    2560 generic.go:356] "Generic (PLEG): container finished" podID="af9a02ed-b575-4b29-9608-c7e5a219149e" containerID="baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8" exitCode=0
Jul 27 21:02:23.907267 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.907131    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"
Jul 27 21:02:23.907267 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.907161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" event={"ID":"af9a02ed-b575-4b29-9608-c7e5a219149e","Type":"ContainerDied","Data":"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"}
Jul 27 21:02:23.907267 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.907203    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb" event={"ID":"af9a02ed-b575-4b29-9608-c7e5a219149e","Type":"ContainerDied","Data":"059dede44c59640a0b32b9c4c15c7d2600f4ea9f88c282ee39b7384ca989f7f7"}
Jul 27 21:02:23.907267 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.907220    2560 scope.go:117] "RemoveContainer" containerID="baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"
Jul 27 21:02:23.916353 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.916343    2560 scope.go:117] "RemoveContainer" containerID="baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"
Jul 27 21:02:23.916601 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:02:23.916581    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8\": container with ID starting with baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8 not found: ID does not exist" containerID="baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"
Jul 27 21:02:23.916664 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.916613    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8"} err="failed to get container status \"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8\": rpc error: code = NotFound desc = could not find container \"baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8\": container with ID starting with baba33681785df2a1dbf4e34792dcbd8dd34e5e776331dc377c5d9a30ca54ae8 not found: ID does not exist"
Jul 27 21:02:23.929353 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.929332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 21:02:23.933142 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:23.933122    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/auth-test-model-132aad-kserve-5946cd67bc-lv7bb"]
Jul 27 21:02:24.709914 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.709882    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:02:24.710332 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710316    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9a02ed-b575-4b29-9608-c7e5a219149e" containerName="main"
Jul 27 21:02:24.710380 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710333    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9a02ed-b575-4b29-9608-c7e5a219149e" containerName="main"
Jul 27 21:02:24.710380 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710346    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8756535-e34c-4b60-ad5f-077fb1fd99cf" containerName="cleanup"
Jul 27 21:02:24.710380 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710353    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8756535-e34c-4b60-ad5f-077fb1fd99cf" containerName="cleanup"
Jul 27 21:02:24.710501 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710424    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9a02ed-b575-4b29-9608-c7e5a219149e" containerName="main"
Jul 27 21:02:24.710501 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.710432    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8756535-e34c-4b60-ad5f-077fb1fd99cf" containerName="cleanup"
Jul 27 21:02:24.713459 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.713428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.717722 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.717701    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"sub-test-model-132aad-kserve-self-signed-certs\""
Jul 27 21:02:24.718971 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.718950    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"default-dockercfg-rvfc5\""
Jul 27 21:02:24.719078 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.718994    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"kube-root-ca.crt\""
Jul 27 21:02:24.719078 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.719004    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-132aad\"/\"openshift-service-ca.crt\""
Jul 27 21:02:24.731533 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.731504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:02:24.750657 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.750919 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs6l\" (UniqueName: \"kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.750919 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.750919 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.750919 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.751086 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.750918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851700 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851700 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851954 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851954 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs6l\" (UniqueName: \"kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851954 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.851954 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.851891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.852223 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.852200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.852295 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.852249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.852295 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.852289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.854085 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.854062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.854287 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.854271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:24.860024 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:24.859997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs6l\" (UniqueName: \"kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l\") pod \"sub-test-model-132aad-kserve-65f85f7f7f-l8wb8\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") " pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:25.023510 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.023425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:25.159451 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.159421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:02:25.161334 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:02:25.161309    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45830e9f_e54a_4938_9f35_1043ec3a4435.slice/crio-f165b2c765c9e6afaadb4cd6c7866532fd088313a36c581ee5e046775379e203 WatchSource:0}: Error finding container f165b2c765c9e6afaadb4cd6c7866532fd088313a36c581ee5e046775379e203: Status 404 returned error can't find the container with id f165b2c765c9e6afaadb4cd6c7866532fd088313a36c581ee5e046775379e203
Jul 27 21:02:25.727046 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.727011    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9a02ed-b575-4b29-9608-c7e5a219149e" path="/var/lib/kubelet/pods/af9a02ed-b575-4b29-9608-c7e5a219149e/volumes"
Jul 27 21:02:25.919523 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.919491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" event={"ID":"45830e9f-e54a-4938-9f35-1043ec3a4435","Type":"ContainerStarted","Data":"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"}
Jul 27 21:02:25.919523 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.919529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" event={"ID":"45830e9f-e54a-4938-9f35-1043ec3a4435","Type":"ContainerStarted","Data":"f165b2c765c9e6afaadb4cd6c7866532fd088313a36c581ee5e046775379e203"}
Jul 27 21:02:25.941584 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:25.941503    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" podStartSLOduration=1.941489129 podStartE2EDuration="1.941489129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:25.940870502 +0000 UTC m=+2678.790637334" watchObservedRunningTime="2026-07-27 21:02:25.941489129 +0000 UTC m=+2678.791256018"
Jul 27 21:02:35.023880 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:35.023842    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:35.024348 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:35.023895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:35.036240 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:35.036217    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:02:35.989333 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:02:35.989293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:03:48.936985 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:03:48.936948    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:04:03.955793 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:03.955746    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" podUID="45830e9f-e54a-4938-9f35-1043ec3a4435" containerName="main" containerID="cri-o://a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2" gracePeriod=45
Jul 27 21:04:04.231057 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.231026    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:04:04.295907 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.295864    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.295907 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.295915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.296166 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.295992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.296166 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296092    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.296166 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.296328 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296168    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjs6l\" (UniqueName: \"kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l\") pod \"45830e9f-e54a-4938-9f35-1043ec3a4435\" (UID: \"45830e9f-e54a-4938-9f35-1043ec3a4435\") "
Jul 27 21:04:04.296328 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache" (OuterVolumeSpecName: "model-cache") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:04:04.296328 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296272    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:04:04.296495 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home" (OuterVolumeSpecName: "home") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:04:04.296607 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296580    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.296607 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296606    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.296803 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.296640    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.298131 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.298105    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:04:04.298216 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.298108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm" (OuterVolumeSpecName: "dshm") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:04:04.298304 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.298289    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l" (OuterVolumeSpecName: "kube-api-access-jjs6l") pod "45830e9f-e54a-4938-9f35-1043ec3a4435" (UID: "45830e9f-e54a-4938-9f35-1043ec3a4435"). InnerVolumeSpecName "kube-api-access-jjs6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:04:04.397970 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.397936    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45830e9f-e54a-4938-9f35-1043ec3a4435-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.397970 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.397962    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjs6l\" (UniqueName: \"kubernetes.io/projected/45830e9f-e54a-4938-9f35-1043ec3a4435-kube-api-access-jjs6l\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.398170 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.397991    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45830e9f-e54a-4938-9f35-1043ec3a4435-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:04:04.433222 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.433190    2560 generic.go:356] "Generic (PLEG): container finished" podID="45830e9f-e54a-4938-9f35-1043ec3a4435" containerID="a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2" exitCode=0
Jul 27 21:04:04.433435 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.433235    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"
Jul 27 21:04:04.433435 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.433243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" event={"ID":"45830e9f-e54a-4938-9f35-1043ec3a4435","Type":"ContainerDied","Data":"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"}
Jul 27 21:04:04.433435 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.433275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8" event={"ID":"45830e9f-e54a-4938-9f35-1043ec3a4435","Type":"ContainerDied","Data":"f165b2c765c9e6afaadb4cd6c7866532fd088313a36c581ee5e046775379e203"}
Jul 27 21:04:04.433435 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.433293    2560 scope.go:117] "RemoveContainer" containerID="a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"
Jul 27 21:04:04.442639 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.442619    2560 scope.go:117] "RemoveContainer" containerID="a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"
Jul 27 21:04:04.442889 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:04:04.442868    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2\": container with ID starting with a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2 not found: ID does not exist" containerID="a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"
Jul 27 21:04:04.442960 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.442896    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2"} err="failed to get container status \"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2\": rpc error: code = NotFound desc = could not find container \"a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2\": container with ID starting with a94347904f1de46c5b2750151d61f77fd19768f71420c97a558f48ac8d7761e2 not found: ID does not exist"
Jul 27 21:04:04.454129 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.454107    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:04:04.456787 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:04.456737    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-132aad/sub-test-model-132aad-kserve-65f85f7f7f-l8wb8"]
Jul 27 21:04:05.727311 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:05.727279    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45830e9f-e54a-4938-9f35-1043ec3a4435" path="/var/lib/kubelet/pods/45830e9f-e54a-4938-9f35-1043ec3a4435/volumes"
Jul 27 21:04:24.590286 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.590250    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:04:24.590989 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.590677    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45830e9f-e54a-4938-9f35-1043ec3a4435" containerName="main"
Jul 27 21:04:24.590989 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.590690    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45830e9f-e54a-4938-9f35-1043ec3a4435" containerName="main"
Jul 27 21:04:24.590989 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.590761    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="45830e9f-e54a-4938-9f35-1043ec3a4435" containerName="main"
Jul 27 21:04:24.593464 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.593445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.596355 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.596329    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66b603\"/\"openshift-service-ca.crt\""
Jul 27 21:04:24.597023 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.596995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66b603\"/\"default-dockercfg-zdpkh\""
Jul 27 21:04:24.598053 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.598021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-66b603\"/\"rate-test-model-66b603-kserve-self-signed-certs\""
Jul 27 21:04:24.598341 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.598319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-66b603\"/\"kube-root-ca.crt\""
Jul 27 21:04:24.609079 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.609051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:04:24.695260 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.695469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.695469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7k8\" (UniqueName: \"kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.695469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.695469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.695469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.695394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.795947 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.795904    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796168 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796168 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7k8\" (UniqueName: \"kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796168 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796168 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796168 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796515 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796574 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.796637 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.796622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.798518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.798488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.798518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.798507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.804945 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.804917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7k8\" (UniqueName: \"kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8\") pod \"rate-test-model-66b603-kserve-55cb65847f-l7m8r\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") " pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:24.905790 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:24.905759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:25.042579 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:25.042548    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:04:25.045430 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:04:25.045401    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e82286_30b2_49a7_9015_13e6bedcc605.slice/crio-684e7f3fd320cb340136ab537b3d41ee6b7f61535cf4adbbad211a4e4e80dbbf WatchSource:0}: Error finding container 684e7f3fd320cb340136ab537b3d41ee6b7f61535cf4adbbad211a4e4e80dbbf: Status 404 returned error can't find the container with id 684e7f3fd320cb340136ab537b3d41ee6b7f61535cf4adbbad211a4e4e80dbbf
Jul 27 21:04:25.540331 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:25.540290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" event={"ID":"06e82286-30b2-49a7-9015-13e6bedcc605","Type":"ContainerStarted","Data":"6fef9e6ea8646f9f35eb544ddd0b770e985181e66090b8c28e244c363766a72e"}
Jul 27 21:04:25.540331 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:25.540345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" event={"ID":"06e82286-30b2-49a7-9015-13e6bedcc605","Type":"ContainerStarted","Data":"684e7f3fd320cb340136ab537b3d41ee6b7f61535cf4adbbad211a4e4e80dbbf"}
Jul 27 21:04:25.557966 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:25.557909    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" podStartSLOduration=1.55789344 podStartE2EDuration="1.55789344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:25.556312254 +0000 UTC m=+2798.406079085" watchObservedRunningTime="2026-07-27 21:04:25.55789344 +0000 UTC m=+2798.407660270"
Jul 27 21:04:34.906217 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:34.906172    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:34.906675 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:34.906256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:34.918867 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:34.918842    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:04:35.602904 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:04:35.602873    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:05:43.791924 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:43.791840    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:05:58.811514 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:58.811461    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" podUID="06e82286-30b2-49a7-9015-13e6bedcc605" containerName="main" containerID="cri-o://6fef9e6ea8646f9f35eb544ddd0b770e985181e66090b8c28e244c363766a72e" gracePeriod=45
Jul 27 21:05:59.009427 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.009395    2560 generic.go:356] "Generic (PLEG): container finished" podID="06e82286-30b2-49a7-9015-13e6bedcc605" containerID="6fef9e6ea8646f9f35eb544ddd0b770e985181e66090b8c28e244c363766a72e" exitCode=0
Jul 27 21:05:59.009579 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.009469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" event={"ID":"06e82286-30b2-49a7-9015-13e6bedcc605","Type":"ContainerDied","Data":"6fef9e6ea8646f9f35eb544ddd0b770e985181e66090b8c28e244c363766a72e"}
Jul 27 21:05:59.068843 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.068771    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:05:59.212101 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212066    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212101 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212307 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212188    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212307 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212307 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212302    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk7k8\" (UniqueName: \"kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212431 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache\") pod \"06e82286-30b2-49a7-9015-13e6bedcc605\" (UID: \"06e82286-30b2-49a7-9015-13e6bedcc605\") "
Jul 27 21:05:59.212431 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home" (OuterVolumeSpecName: "home") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:59.212535 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:59.212652 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache" (OuterVolumeSpecName: "model-cache") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:59.212714 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212684    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-tmp-dir\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:05:59.212714 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.212697    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-home\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:05:59.214505 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.214485    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:05:59.214611 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.214522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm" (OuterVolumeSpecName: "dshm") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:05:59.214611 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.214567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8" (OuterVolumeSpecName: "kube-api-access-fk7k8") pod "06e82286-30b2-49a7-9015-13e6bedcc605" (UID: "06e82286-30b2-49a7-9015-13e6bedcc605"). InnerVolumeSpecName "kube-api-access-fk7k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:05:59.314113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.314076    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7k8\" (UniqueName: \"kubernetes.io/projected/06e82286-30b2-49a7-9015-13e6bedcc605-kube-api-access-fk7k8\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:05:59.314113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.314115    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-model-cache\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:05:59.314314 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.314128    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e82286-30b2-49a7-9015-13e6bedcc605-tls-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:05:59.314314 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:05:59.314139    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e82286-30b2-49a7-9015-13e6bedcc605-dshm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:06:00.015181 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.015098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r" event={"ID":"06e82286-30b2-49a7-9015-13e6bedcc605","Type":"ContainerDied","Data":"684e7f3fd320cb340136ab537b3d41ee6b7f61535cf4adbbad211a4e4e80dbbf"}
Jul 27 21:06:00.015181 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.015149    2560 scope.go:117] "RemoveContainer" containerID="6fef9e6ea8646f9f35eb544ddd0b770e985181e66090b8c28e244c363766a72e"
Jul 27 21:06:00.015697 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.015107    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"
Jul 27 21:06:00.035791 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.035764    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:06:00.037523 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.037497    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140101039111600"]
Jul 27 21:06:00.037949 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.037932    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e82286-30b2-49a7-9015-13e6bedcc605" containerName="main"
Jul 27 21:06:00.038039 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.037951    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e82286-30b2-49a7-9015-13e6bedcc605" containerName="main"
Jul 27 21:06:00.038095 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.038064    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e82286-30b2-49a7-9015-13e6bedcc605" containerName="main"
Jul 27 21:06:00.042764 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.042650    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-66b603/rate-test-model-66b603-kserve-55cb65847f-l7m8r"]
Jul 27 21:06:00.042764 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.042756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:00.044962 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.044936    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140101039111600"]
Jul 27 21:06:00.045492 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.045471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hmb6d\""
Jul 27 21:06:00.119782 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.119754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcbh\" (UniqueName: \"kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh\") pod \"test-curl-7612-140101039111600\" (UID: \"776b46cd-8ea1-415c-8274-1cd3971a938e\") " pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:00.220388 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.220359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcbh\" (UniqueName: \"kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh\") pod \"test-curl-7612-140101039111600\" (UID: \"776b46cd-8ea1-415c-8274-1cd3971a938e\") " pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:00.228563 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.228537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcbh\" (UniqueName: \"kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh\") pod \"test-curl-7612-140101039111600\" (UID: \"776b46cd-8ea1-415c-8274-1cd3971a938e\") " pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:00.353008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.352920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:00.483550 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.483521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140101039111600"]
Jul 27 21:06:00.485154 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:06:00.485130    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776b46cd_8ea1_415c_8274_1cd3971a938e.slice/crio-fa0b752a0ba5207645ce75f90bad5b535f02b204f6b2ab805f8f8f869086753d WatchSource:0}: Error finding container fa0b752a0ba5207645ce75f90bad5b535f02b204f6b2ab805f8f8f869086753d: Status 404 returned error can't find the container with id fa0b752a0ba5207645ce75f90bad5b535f02b204f6b2ab805f8f8f869086753d
Jul 27 21:06:00.486456 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:00.486436    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:06:01.022436 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:01.022398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140101039111600" event={"ID":"776b46cd-8ea1-415c-8274-1cd3971a938e","Type":"ContainerStarted","Data":"fa0b752a0ba5207645ce75f90bad5b535f02b204f6b2ab805f8f8f869086753d"}
Jul 27 21:06:01.726469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:01.726432    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e82286-30b2-49a7-9015-13e6bedcc605" path="/var/lib/kubelet/pods/06e82286-30b2-49a7-9015-13e6bedcc605/volumes"
Jul 27 21:06:02.029394 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:02.029306    2560 generic.go:356] "Generic (PLEG): container finished" podID="776b46cd-8ea1-415c-8274-1cd3971a938e" containerID="43fbc387f6955de4a16cd9392d0c78a32b458a8979f2bdbcd05d50acbd40c651" exitCode=0
Jul 27 21:06:02.029753 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:02.029385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140101039111600" event={"ID":"776b46cd-8ea1-415c-8274-1cd3971a938e","Type":"ContainerDied","Data":"43fbc387f6955de4a16cd9392d0c78a32b458a8979f2bdbcd05d50acbd40c651"}
Jul 27 21:06:03.165090 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.165066    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:03.200673 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.200634    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140101039111600_776b46cd-8ea1-415c-8274-1cd3971a938e/test-curl-7612-140101039111600/0.log"
Jul 27 21:06:03.223143 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.223114    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140101039111600"]
Jul 27 21:06:03.226548 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.226526    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140101039111600"]
Jul 27 21:06:03.249987 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.249963    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xcbh\" (UniqueName: \"kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh\") pod \"776b46cd-8ea1-415c-8274-1cd3971a938e\" (UID: \"776b46cd-8ea1-415c-8274-1cd3971a938e\") "
Jul 27 21:06:03.252017 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.251993    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh" (OuterVolumeSpecName: "kube-api-access-5xcbh") pod "776b46cd-8ea1-415c-8274-1cd3971a938e" (UID: "776b46cd-8ea1-415c-8274-1cd3971a938e"). InnerVolumeSpecName "kube-api-access-5xcbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:03.351672 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.351586    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcbh\" (UniqueName: \"kubernetes.io/projected/776b46cd-8ea1-415c-8274-1cd3971a938e-kube-api-access-5xcbh\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:06:03.726504 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.726470    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776b46cd-8ea1-415c-8274-1cd3971a938e" path="/var/lib/kubelet/pods/776b46cd-8ea1-415c-8274-1cd3971a938e/volumes"
Jul 27 21:06:03.977636 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.977552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:06:03.978146 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.978109    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="776b46cd-8ea1-415c-8274-1cd3971a938e" containerName="test-curl-7612-140101039111600"
Jul 27 21:06:03.978146 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.978137    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="776b46cd-8ea1-415c-8274-1cd3971a938e" containerName="test-curl-7612-140101039111600"
Jul 27 21:06:03.978313 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.978238    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="776b46cd-8ea1-415c-8274-1cd3971a938e" containerName="test-curl-7612-140101039111600"
Jul 27 21:06:03.981315 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.981295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:03.984352 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.984330    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6d6a46-openshift-default-dockercfg-shkzh\""
Jul 27 21:06:03.996302 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:03.996269    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:06:04.041802 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.041773    2560 scope.go:117] "RemoveContainer" containerID="43fbc387f6955de4a16cd9392d0c78a32b458a8979f2bdbcd05d50acbd40c651"
Jul 27 21:06:04.042002 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.041830    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140101039111600"
Jul 27 21:06:04.159664 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.159664 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.159910 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.159910 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.159910 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.159910 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhkl\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.160072 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.160072 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.159983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.160072 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.160044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261084 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.260993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261084 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261084 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261084 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261084 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhkl\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.261801 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.262113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.262113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.261887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.263769 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.263750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.263913 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.263895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.269847 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.269779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhkl\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.269847 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.269831    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token\") pod \"e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") " pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.291855 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.291829    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:04.431559 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.431535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:06:04.433206 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:06:04.433159    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09039a24_aa09_427e_82cb_c3365de91622.slice/crio-08af3b01bcd868238a2d883fdd952237fbfcb671c1f05f3ecaf59f26f031760c WatchSource:0}: Error finding container 08af3b01bcd868238a2d883fdd952237fbfcb671c1f05f3ecaf59f26f031760c: Status 404 returned error can't find the container with id 08af3b01bcd868238a2d883fdd952237fbfcb671c1f05f3ecaf59f26f031760c
Jul 27 21:06:04.435597 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.435550    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:04.435693 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.435646    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:04.435736 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:04.435691    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:05.048539 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:05.048502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerStarted","Data":"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"}
Jul 27 21:06:05.048776 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:05.048558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerStarted","Data":"08af3b01bcd868238a2d883fdd952237fbfcb671c1f05f3ecaf59f26f031760c"}
Jul 27 21:06:05.071486 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:05.071434    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podStartSLOduration=2.071419547 podStartE2EDuration="2.071419547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:05.06933572 +0000 UTC m=+2897.919102548" watchObservedRunningTime="2026-07-27 21:06:05.071419547 +0000 UTC m=+2897.921186377"
Jul 27 21:06:05.292402 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:05.292371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:05.297441 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:05.297416    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:06.054663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:06.054634    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:06.055560 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:06.055541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:22.096953 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.096907    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 21:06:22.097364 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.096991    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 21:06:22.620298 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.620262    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:06:22.624342 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.624321    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.626885 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.626861    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cae454-openshift-default-dockercfg-74jj5\""
Jul 27 21:06:22.635909 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.635886    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:06:22.744649 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.744804 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.744804 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.744930 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.744930 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.745008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzcl\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.745008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.745008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.744995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.745114 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.745013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.845798 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.845763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.845798 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.845802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846045 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.845842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzcl\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846045 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.845891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846273 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846273 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846370 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846370 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846512 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846680 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846823 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.846908 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.846862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.848422 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.848399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.848524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.848496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.854464 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.854438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzcl\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.854560 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.854490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token\") pod \"e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") " pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:22.936588 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:22.936491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:23.090268 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:23.090232    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:06:23.093668 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:06:23.093636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc271fe45_1ab2_4f78_8b87_05232627bc64.slice/crio-c2f8429bec5600fde4f8fff8ae31dbe51778c6a779a55439cb6ed34f15780a7e WatchSource:0}: Error finding container c2f8429bec5600fde4f8fff8ae31dbe51778c6a779a55439cb6ed34f15780a7e: Status 404 returned error can't find the container with id c2f8429bec5600fde4f8fff8ae31dbe51778c6a779a55439cb6ed34f15780a7e
Jul 27 21:06:23.096341 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:23.096297    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:23.096424 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:23.096395    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:23.096465 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:23.096445    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:23.145404 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:23.145376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerStarted","Data":"c2f8429bec5600fde4f8fff8ae31dbe51778c6a779a55439cb6ed34f15780a7e"}
Jul 27 21:06:24.151156 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.151118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerStarted","Data":"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"}
Jul 27 21:06:24.152609 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.152584    2560 generic.go:356] "Generic (PLEG): container finished" podID="09039a24-aa09-427e-82cb-c3365de91622" containerID="cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee" exitCode=0
Jul 27 21:06:24.152727 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.152614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerDied","Data":"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"}
Jul 27 21:06:24.152991 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.152975    2560 scope.go:117] "RemoveContainer" containerID="cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"
Jul 27 21:06:24.154029 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.154000    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:24.154085 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.154070    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:24.154124 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.154105    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:24.173890 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.173839    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podStartSLOduration=2.17379971 podStartE2EDuration="2.17379971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:24.171852495 +0000 UTC m=+2917.021619327" watchObservedRunningTime="2026-07-27 21:06:24.17379971 +0000 UTC m=+2917.023566555"
Jul 27 21:06:24.937294 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.937260    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:24.942142 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:24.942118    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:25.165116 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.165070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerStarted","Data":"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"}
Jul 27 21:06:25.165598 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.165554    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:25.166901 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.166876    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:25.292269 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.292159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:25.292269 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.292247    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 27 21:06:25.292511 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:25.292328    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 27 21:06:26.293021 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:26.292986    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 27 21:06:26.293426 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:26.293054    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 27 21:06:27.292583 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:27.292535    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 27 21:06:27.292803 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:27.292613    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 27 21:06:28.293222 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:28.293181    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 27 21:06:28.293628 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:28.293236    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 27 21:06:29.292821 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:29.292777    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Jul 27 21:06:29.293047 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:29.292870    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Jul 27 21:06:30.296255 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:30.296224    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:30.296667 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:30.296615    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:30.297366 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:30.297351    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:06:44.271903 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.271868    2560 generic.go:356] "Generic (PLEG): container finished" podID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerID="c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95" exitCode=0
Jul 27 21:06:44.272514 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.271928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerDied","Data":"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"}
Jul 27 21:06:44.272514 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.272461    2560 scope.go:117] "RemoveContainer" containerID="c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"
Jul 27 21:06:44.273867 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.273799    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:44.273975 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.273918    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:44.273975 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:44.273965    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:06:45.283627 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:45.282480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerStarted","Data":"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"}
Jul 27 21:06:45.937642 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:45.937601    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:45.937887 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:45.937860    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Jul 27 21:06:45.937949 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:45.937916    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Jul 27 21:06:46.937881 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:46.937844    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Jul 27 21:06:46.938309 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:46.937910    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Jul 27 21:06:47.937294 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:47.937256    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Jul 27 21:06:47.937450 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:47.937315    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Jul 27 21:06:48.937590 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:48.937555    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Jul 27 21:06:48.938036 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:48.937612    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Jul 27 21:06:49.941120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:49.941089    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:49.941584 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:49.941522    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:06:49.942108 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:06:49.942090    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:07:48.622606 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.622572    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:07:48.626285 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.626268    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.629158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.629138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1482c3-openshift-default-dockercfg-bcx8c\""
Jul 27 21:07:48.639740 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.639714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:07:48.718766 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.718968 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.718968 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.718968 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.718968 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.719491 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.718989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.719491 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.719039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprcv\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.719491 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.719059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.719491 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.719083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820268 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprcv\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820753 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820753 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820753 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.820970 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.820754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.821054 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.821029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.821117 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.821080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.821180 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.821111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.821347 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.821327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.822608 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.822590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.822876 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.822860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.839275 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.839253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.843193 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.843176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprcv\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv\") pod \"e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") " pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:48.936451 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:48.936358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:49.079025 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.078999    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:07:49.080364 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:07:49.080337    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446b3ac9_213f_4494_80c5_5ccbb8e43431.slice/crio-9279fbf90c52e5ab1c1b8c97c1f0b689b58a62d4fbff0c534dbb174da659c7b8 WatchSource:0}: Error finding container 9279fbf90c52e5ab1c1b8c97c1f0b689b58a62d4fbff0c534dbb174da659c7b8: Status 404 returned error can't find the container with id 9279fbf90c52e5ab1c1b8c97c1f0b689b58a62d4fbff0c534dbb174da659c7b8
Jul 27 21:07:49.082483 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.082445    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:07:49.082599 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.082561    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:07:49.082599 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.082593    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:07:49.616501 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.616468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerStarted","Data":"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"}
Jul 27 21:07:49.617037 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.616523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerStarted","Data":"9279fbf90c52e5ab1c1b8c97c1f0b689b58a62d4fbff0c534dbb174da659c7b8"}
Jul 27 21:07:49.667053 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.666992    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podStartSLOduration=1.66697615 podStartE2EDuration="1.66697615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:49.664265574 +0000 UTC m=+3002.514032406" watchObservedRunningTime="2026-07-27 21:07:49.66697615 +0000 UTC m=+3002.516742981"
Jul 27 21:07:49.937149 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:49.937075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:50.942451 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:50.942421    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:51.626908 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:51.626871    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:07:51.628385 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:07:51.628363    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:06.628164 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.628123    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Jul 27 21:08:06.628617 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.628189    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Jul 27 21:08:06.721690 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.721661    2560 generic.go:356] "Generic (PLEG): container finished" podID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerID="f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8" exitCode=0
Jul 27 21:08:06.721891 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.721730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerDied","Data":"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"}
Jul 27 21:08:06.722082 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.722065    2560 scope.go:117] "RemoveContainer" containerID="f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"
Jul 27 21:08:06.723043 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.723017    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:08:06.723107 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.723080    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:08:06.723158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:06.723113    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 21:08:07.729100 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:07.729070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerStarted","Data":"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"}
Jul 27 21:08:07.936984 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:07.936945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:07.937184 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:07.937160    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Jul 27 21:08:07.937247 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:07.937215    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Jul 27 21:08:08.937331 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:08.937293    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Jul 27 21:08:08.937744 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:08.937359    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Jul 27 21:08:09.937164 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:09.937130    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Jul 27 21:08:09.937362 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:09.937199    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Jul 27 21:08:10.937663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:10.937620    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Jul 27 21:08:10.938088 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:10.937679    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Jul 27 21:08:11.940898 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:11.940858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:11.941327 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:11.941140    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:11.941870 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:11.941849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:28.508501 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:28.508467    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:08:28.509143 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:28.508705    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy" containerID="cri-o://425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e" gracePeriod=30
Jul 27 21:08:33.763665 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.763634    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:33.777185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777156    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777334 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777192    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tprcv\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777334 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777334 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777269    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777334 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777306    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777552 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777356    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777552 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777396    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777552 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy\") pod \"446b3ac9-213f-4494-80c5-5ccbb8e43431\" (UID: \"446b3ac9-213f-4494-80c5-5ccbb8e43431\") "
Jul 27 21:08:33.777853 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:08:33.777853 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:08:33.777853 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777791    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:08:33.777853 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:08:33.778120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777952    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.778120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777972    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/446b3ac9-213f-4494-80c5-5ccbb8e43431-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.778120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.777986    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.778120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.778001    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.778344 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.778293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data" (OuterVolumeSpecName: "istio-data") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:08:33.779673 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.779649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:08:33.779761 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.779655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:08:33.779914 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.779887    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv" (OuterVolumeSpecName: "kube-api-access-tprcv") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "kube-api-access-tprcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:08:33.779914 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.779904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token" (OuterVolumeSpecName: "istio-token") pod "446b3ac9-213f-4494-80c5-5ccbb8e43431" (UID: "446b3ac9-213f-4494-80c5-5ccbb8e43431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:08:33.864024 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.863992    2560 generic.go:356] "Generic (PLEG): container finished" podID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerID="425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e" exitCode=0
Jul 27 21:08:33.864229 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.864040    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"
Jul 27 21:08:33.864229 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.864087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerDied","Data":"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"}
Jul 27 21:08:33.864229 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.864148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd" event={"ID":"446b3ac9-213f-4494-80c5-5ccbb8e43431","Type":"ContainerDied","Data":"9279fbf90c52e5ab1c1b8c97c1f0b689b58a62d4fbff0c534dbb174da659c7b8"}
Jul 27 21:08:33.864229 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.864171    2560 scope.go:117] "RemoveContainer" containerID="425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"
Jul 27 21:08:33.879361 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.879334    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.879459 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.879365    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprcv\" (UniqueName: \"kubernetes.io/projected/446b3ac9-213f-4494-80c5-5ccbb8e43431-kube-api-access-tprcv\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.879459 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.879377    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.879459 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.879386    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.879459 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.879393    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/446b3ac9-213f-4494-80c5-5ccbb8e43431-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:08:33.880109 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.880083    2560 scope.go:117] "RemoveContainer" containerID="f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"
Jul 27 21:08:33.888353 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.888331    2560 scope.go:117] "RemoveContainer" containerID="425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"
Jul 27 21:08:33.888619 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:08:33.888598    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e\": container with ID starting with 425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e not found: ID does not exist" containerID="425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"
Jul 27 21:08:33.888672 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.888631    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e"} err="failed to get container status \"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e\": rpc error: code = NotFound desc = could not find container \"425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e\": container with ID starting with 425253fa6029e527663ed9156b2c6f688aeb2a95a757ec9f1dab468a38fc783e not found: ID does not exist"
Jul 27 21:08:33.888672 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.888651    2560 scope.go:117] "RemoveContainer" containerID="f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"
Jul 27 21:08:33.888870 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:08:33.888856    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8\": container with ID starting with f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8 not found: ID does not exist" containerID="f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"
Jul 27 21:08:33.888923 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.888873    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8"} err="failed to get container status \"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8\": rpc error: code = NotFound desc = could not find container \"f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8\": container with ID starting with f6e532c82306f4e82c53ff552ad92b3974ec84bc4dbe8b017476a824e36e97c8 not found: ID does not exist"
Jul 27 21:08:33.893088 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.893068    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:08:33.896887 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:33.896866    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1482c3-openshift-default-c4f685fcd-c6lbd"]
Jul 27 21:08:35.726389 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:08:35.726356    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" path="/var/lib/kubelet/pods/446b3ac9-213f-4494-80c5-5ccbb8e43431/volumes"
Jul 27 21:09:40.847169 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:40.847135    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:09:40.847627 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:40.847341    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" containerID="cri-o://34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5" gracePeriod=30
Jul 27 21:09:45.299463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:45.299426    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:09:45.299911 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:45.299492    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:09:46.108479 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.108419    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:09:46.190244 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190244 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190250    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190271    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qhkl\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190306    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190375    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190462    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190524 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190510    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo\") pod \"09039a24-aa09-427e-82cb-c3365de91622\" (UID: \"09039a24-aa09-427e-82cb-c3365de91622\") "
Jul 27 21:09:46.190938 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:46.190938 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:46.190938 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:09:46.190938 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:46.191138 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190947    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09039a24-aa09-427e-82cb-c3365de91622-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.191138 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190968    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.191138 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.190981    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.191287 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.191222    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data" (OuterVolumeSpecName: "istio-data") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:46.192632 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.192602    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:09:46.192775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.192647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token" (OuterVolumeSpecName: "istio-token") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:09:46.193042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.193023    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:09:46.193128 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.193046    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl" (OuterVolumeSpecName: "kube-api-access-7qhkl") pod "09039a24-aa09-427e-82cb-c3365de91622" (UID: "09039a24-aa09-427e-82cb-c3365de91622"). InnerVolumeSpecName "kube-api-access-7qhkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:09:46.243376 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.243347    2560 generic.go:356] "Generic (PLEG): container finished" podID="09039a24-aa09-427e-82cb-c3365de91622" containerID="34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5" exitCode=0
Jul 27 21:09:46.243541 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.243401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"
Jul 27 21:09:46.243541 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.243428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerDied","Data":"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"}
Jul 27 21:09:46.243541 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.243491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm" event={"ID":"09039a24-aa09-427e-82cb-c3365de91622","Type":"ContainerDied","Data":"08af3b01bcd868238a2d883fdd952237fbfcb671c1f05f3ecaf59f26f031760c"}
Jul 27 21:09:46.243541 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.243514    2560 scope.go:117] "RemoveContainer" containerID="34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"
Jul 27 21:09:46.253587 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.253571    2560 scope.go:117] "RemoveContainer" containerID="cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"
Jul 27 21:09:46.261855 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.261832    2560 scope.go:117] "RemoveContainer" containerID="34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"
Jul 27 21:09:46.262117 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:09:46.262098    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5\": container with ID starting with 34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5 not found: ID does not exist" containerID="34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"
Jul 27 21:09:46.262162 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.262125    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5"} err="failed to get container status \"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5\": rpc error: code = NotFound desc = could not find container \"34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5\": container with ID starting with 34383ed83b7d702efdac1c0662298867283e7bc14e7f72c68115515c68226ea5 not found: ID does not exist"
Jul 27 21:09:46.262162 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.262142    2560 scope.go:117] "RemoveContainer" containerID="cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"
Jul 27 21:09:46.262406 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:09:46.262388    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee\": container with ID starting with cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee not found: ID does not exist" containerID="cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"
Jul 27 21:09:46.262445 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.262412    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee"} err="failed to get container status \"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee\": rpc error: code = NotFound desc = could not find container \"cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee\": container with ID starting with cc6a31cfe8c2dfc595e013e06972b9ace9acf482f50b7714aa14888162562eee not found: ID does not exist"
Jul 27 21:09:46.271239 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.271218    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:09:46.276735 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.276714    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6d6a46-openshift-default-d8b8cf59b-pvgcm"]
Jul 27 21:09:46.292378 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292352    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.292378 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292375    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.292506 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292385    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.292506 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292394    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09039a24-aa09-427e-82cb-c3365de91622-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.292506 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292402    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09039a24-aa09-427e-82cb-c3365de91622-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:46.292506 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:46.292409    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhkl\" (UniqueName: \"kubernetes.io/projected/09039a24-aa09-427e-82cb-c3365de91622-kube-api-access-7qhkl\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:09:47.732370 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:09:47.732335    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09039a24-aa09-427e-82cb-c3365de91622" path="/var/lib/kubelet/pods/09039a24-aa09-427e-82cb-c3365de91622/volumes"
Jul 27 21:10:02.944599 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:02.944560    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:10:02.945189 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:02.944865    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" containerID="cri-o://6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3" gracePeriod=30
Jul 27 21:10:04.944335 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:04.944290    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:10:04.944729 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:04.944354    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:10:08.191621 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.191598    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:10:08.301588 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301588 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301588 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301556    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301650    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wzcl\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301695    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301741    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.301894 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301848    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token\") pod \"c271fe45-1ab2-4f78-8b87-05232627bc64\" (UID: \"c271fe45-1ab2-4f78-8b87-05232627bc64\") "
Jul 27 21:10:08.302194 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.301947    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:08.302194 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302023    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:10:08.302194 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302034    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:08.302194 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:08.302382 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302260    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.302382 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302274    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.302382 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302283    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.302382 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302291    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c271fe45-1ab2-4f78-8b87-05232627bc64-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.302895 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.302870    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data" (OuterVolumeSpecName: "istio-data") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:08.304082 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.304061    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token" (OuterVolumeSpecName: "istio-token") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:08.304195 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.304075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl" (OuterVolumeSpecName: "kube-api-access-2wzcl") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "kube-api-access-2wzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:08.304243 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.304210    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:08.304243 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.304221    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c271fe45-1ab2-4f78-8b87-05232627bc64" (UID: "c271fe45-1ab2-4f78-8b87-05232627bc64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:10:08.360063 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.359040    2560 generic.go:356] "Generic (PLEG): container finished" podID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerID="6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3" exitCode=0
Jul 27 21:10:08.360063 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.359092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerDied","Data":"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"}
Jul 27 21:10:08.360063 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.359190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt" event={"ID":"c271fe45-1ab2-4f78-8b87-05232627bc64","Type":"ContainerDied","Data":"c2f8429bec5600fde4f8fff8ae31dbe51778c6a779a55439cb6ed34f15780a7e"}
Jul 27 21:10:08.360063 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.359230    2560 scope.go:117] "RemoveContainer" containerID="6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"
Jul 27 21:10:08.360063 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.359457    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"
Jul 27 21:10:08.372349 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.372328    2560 scope.go:117] "RemoveContainer" containerID="c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"
Jul 27 21:10:08.380725 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.380705    2560 scope.go:117] "RemoveContainer" containerID="6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"
Jul 27 21:10:08.381039 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:08.381015    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3\": container with ID starting with 6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3 not found: ID does not exist" containerID="6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"
Jul 27 21:10:08.381106 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.381049    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3"} err="failed to get container status \"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3\": rpc error: code = NotFound desc = could not find container \"6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3\": container with ID starting with 6cdf986b91eed5789deef2c952a4a8999391127f2aa8c866477d78d80fc59ab3 not found: ID does not exist"
Jul 27 21:10:08.381106 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.381070    2560 scope.go:117] "RemoveContainer" containerID="c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"
Jul 27 21:10:08.381319 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:08.381300    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95\": container with ID starting with c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95 not found: ID does not exist" containerID="c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"
Jul 27 21:10:08.381365 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.381329    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95"} err="failed to get container status \"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95\": rpc error: code = NotFound desc = could not find container \"c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95\": container with ID starting with c8ab3b0aa247011e1bad6f636f7ba7f61cb83e89958fd3ebe0dd3092e18a4d95 not found: ID does not exist"
Jul 27 21:10:08.385702 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.385679    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:10:08.389900 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.389877    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cae454-openshift-default-f88bb4bcb-npvpt"]
Jul 27 21:10:08.403156 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.403138    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.403156 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.403156    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.403276 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.403164    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.403276 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.403171    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c271fe45-1ab2-4f78-8b87-05232627bc64-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:08.403276 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:08.403179    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzcl\" (UniqueName: \"kubernetes.io/projected/c271fe45-1ab2-4f78-8b87-05232627bc64-kube-api-access-2wzcl\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:09.726669 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:09.726591    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" path="/var/lib/kubelet/pods/c271fe45-1ab2-4f78-8b87-05232627bc64/volumes"
Jul 27 21:10:13.778744 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:13.778699    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 21:10:13.779186 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:13.778924    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" podUID="a1682669-bd42-4fc3-9faa-5f9043567482" containerName="maas-api" containerID="cri-o://3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1" gracePeriod=30
Jul 27 21:10:14.035161 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.035095    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 21:10:14.158983 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.158950    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca\") pod \"a1682669-bd42-4fc3-9faa-5f9043567482\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") "
Jul 27 21:10:14.159180 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.159023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls\") pod \"a1682669-bd42-4fc3-9faa-5f9043567482\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") "
Jul 27 21:10:14.159180 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.159114    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfjff\" (UniqueName: \"kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff\") pod \"a1682669-bd42-4fc3-9faa-5f9043567482\" (UID: \"a1682669-bd42-4fc3-9faa-5f9043567482\") "
Jul 27 21:10:14.159393 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.159361    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1682669-bd42-4fc3-9faa-5f9043567482" (UID: "a1682669-bd42-4fc3-9faa-5f9043567482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:10:14.161160 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.161137    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1682669-bd42-4fc3-9faa-5f9043567482" (UID: "a1682669-bd42-4fc3-9faa-5f9043567482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:10:14.161758 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.161734    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff" (OuterVolumeSpecName: "kube-api-access-kfjff") pod "a1682669-bd42-4fc3-9faa-5f9043567482" (UID: "a1682669-bd42-4fc3-9faa-5f9043567482"). InnerVolumeSpecName "kube-api-access-kfjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:14.260016 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.259982    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjff\" (UniqueName: \"kubernetes.io/projected/a1682669-bd42-4fc3-9faa-5f9043567482-kube-api-access-kfjff\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:14.260016 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.260008    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1682669-bd42-4fc3-9faa-5f9043567482-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:14.260016 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.260018    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1682669-bd42-4fc3-9faa-5f9043567482-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:14.389350 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.389317    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1682669-bd42-4fc3-9faa-5f9043567482" containerID="3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1" exitCode=0
Jul 27 21:10:14.389555 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.389360    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"
Jul 27 21:10:14.389555 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.389402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" event={"ID":"a1682669-bd42-4fc3-9faa-5f9043567482","Type":"ContainerDied","Data":"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"}
Jul 27 21:10:14.389555 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.389453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2" event={"ID":"a1682669-bd42-4fc3-9faa-5f9043567482","Type":"ContainerDied","Data":"961228911a43b7c5e53c95de142d579087ee14f36dbfa8143fb3d0cd21f6ef95"}
Jul 27 21:10:14.389555 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.389484    2560 scope.go:117] "RemoveContainer" containerID="3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"
Jul 27 21:10:14.398597 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.398580    2560 scope.go:117] "RemoveContainer" containerID="3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"
Jul 27 21:10:14.398883 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:14.398865    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1\": container with ID starting with 3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1 not found: ID does not exist" containerID="3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"
Jul 27 21:10:14.398943 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.398892    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1"} err="failed to get container status \"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1\": rpc error: code = NotFound desc = could not find container \"3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1\": container with ID starting with 3ef01cf75d4909ec1ab200573a731baff79f66c0969cb402c86c2da7177fcad1 not found: ID does not exist"
Jul 27 21:10:14.410684 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.410662    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 21:10:14.414774 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:14.414755    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-132aad-7854b546cf-gg2q2"]
Jul 27 21:10:15.726109 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:15.726075    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1682669-bd42-4fc3-9faa-5f9043567482" path="/var/lib/kubelet/pods/a1682669-bd42-4fc3-9faa-5f9043567482/volumes"
Jul 27 21:10:24.818742 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:24.818695    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 21:10:24.819274 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:24.818943    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy" containerID="cri-o://bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34" gracePeriod=30
Jul 27 21:10:30.070682 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.070657    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 21:10:30.115156 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115318 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115318 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115318 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115252    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzldg\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115318 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115543 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115433    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115543 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115656 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115562    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:30.115656 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115656 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket\") pod \"4cdc1096-182c-4115-99a1-e49df98662c3\" (UID: \"4cdc1096-182c-4115-99a1-e49df98662c3\") "
Jul 27 21:10:30.115798 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115649    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:10:30.116008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115933    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:30.116008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115938    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cdc1096-182c-4115-99a1-e49df98662c3-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.116008 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.115976    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.116282 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.116119    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:30.116446 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.116427    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:30.117574 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.117529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg" (OuterVolumeSpecName: "kube-api-access-jzldg") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "kube-api-access-jzldg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:30.117756 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.117739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:30.117900 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.117885    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:10:30.118062 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.118039    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "4cdc1096-182c-4115-99a1-e49df98662c3" (UID: "4cdc1096-182c-4115-99a1-e49df98662c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:30.216417 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216386    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216417 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216413    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216417 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216422    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216417 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216430    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216674 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216446    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cdc1096-182c-4115-99a1-e49df98662c3-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216674 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216454    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzldg\" (UniqueName: \"kubernetes.io/projected/4cdc1096-182c-4115-99a1-e49df98662c3-kube-api-access-jzldg\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.216674 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.216463    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cdc1096-182c-4115-99a1-e49df98662c3-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:30.471042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.470890    2560 generic.go:356] "Generic (PLEG): container finished" podID="4cdc1096-182c-4115-99a1-e49df98662c3" containerID="bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34" exitCode=0
Jul 27 21:10:30.471042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.470943    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"
Jul 27 21:10:30.471042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.470948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerDied","Data":"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"}
Jul 27 21:10:30.471042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.471002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9" event={"ID":"4cdc1096-182c-4115-99a1-e49df98662c3","Type":"ContainerDied","Data":"7e3e5dd23e3a045b70730fb0ea8051640cc77f2782f77365eb179e0681243729"}
Jul 27 21:10:30.471042 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.471030    2560 scope.go:117] "RemoveContainer" containerID="bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"
Jul 27 21:10:30.481612 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.481592    2560 scope.go:117] "RemoveContainer" containerID="fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"
Jul 27 21:10:30.493620 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.493595    2560 scope.go:117] "RemoveContainer" containerID="bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"
Jul 27 21:10:30.494316 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:30.494291    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34\": container with ID starting with bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34 not found: ID does not exist" containerID="bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"
Jul 27 21:10:30.494433 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.494326    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34"} err="failed to get container status \"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34\": rpc error: code = NotFound desc = could not find container \"bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34\": container with ID starting with bc1374b484b9d7b509f2719dbe0363173f674ca42e2c798f7eaa3b917a062e34 not found: ID does not exist"
Jul 27 21:10:30.494433 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.494352    2560 scope.go:117] "RemoveContainer" containerID="fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"
Jul 27 21:10:30.494676 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:30.494653    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729\": container with ID starting with fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729 not found: ID does not exist" containerID="fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"
Jul 27 21:10:30.494760 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.494684    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729"} err="failed to get container status \"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729\": rpc error: code = NotFound desc = could not find container \"fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729\": container with ID starting with fd38a0004896a495c89914a1c1a930385d041b7924c638e3e11d987ad9805729 not found: ID does not exist"
Jul 27 21:10:30.497018 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.496998    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 21:10:30.498664 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:30.498645    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-132aad-openshift-default-5dc55d89c6-7ndb9"]
Jul 27 21:10:31.725921 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:31.725892    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" path="/var/lib/kubelet/pods/4cdc1096-182c-4115-99a1-e49df98662c3/volumes"
Jul 27 21:10:35.617651 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.617613    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 21:10:35.618117 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.617918    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" podUID="3299c8d5-27d2-4645-900a-837a2f2e42b5" containerName="maas-api" containerID="cri-o://0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547" gracePeriod=30
Jul 27 21:10:35.883355 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.883328    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 21:10:35.967706 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.967663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca\") pod \"3299c8d5-27d2-4645-900a-837a2f2e42b5\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") "
Jul 27 21:10:35.967706 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.967704    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9pnn\" (UniqueName: \"kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn\") pod \"3299c8d5-27d2-4645-900a-837a2f2e42b5\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") "
Jul 27 21:10:35.967974 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.967789    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls\") pod \"3299c8d5-27d2-4645-900a-837a2f2e42b5\" (UID: \"3299c8d5-27d2-4645-900a-837a2f2e42b5\") "
Jul 27 21:10:35.968093 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.968069    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3299c8d5-27d2-4645-900a-837a2f2e42b5" (UID: "3299c8d5-27d2-4645-900a-837a2f2e42b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:10:35.970037 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.970008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn" (OuterVolumeSpecName: "kube-api-access-q9pnn") pod "3299c8d5-27d2-4645-900a-837a2f2e42b5" (UID: "3299c8d5-27d2-4645-900a-837a2f2e42b5"). InnerVolumeSpecName "kube-api-access-q9pnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:35.970037 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:35.970025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3299c8d5-27d2-4645-900a-837a2f2e42b5" (UID: "3299c8d5-27d2-4645-900a-837a2f2e42b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:10:36.068762 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.068734    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3299c8d5-27d2-4645-900a-837a2f2e42b5-openshift-service-ca\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:36.068762 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.068760    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9pnn\" (UniqueName: \"kubernetes.io/projected/3299c8d5-27d2-4645-900a-837a2f2e42b5-kube-api-access-q9pnn\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:36.068762 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.068769    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3299c8d5-27d2-4645-900a-837a2f2e42b5-maas-api-tls\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:36.503000 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.502965    2560 generic.go:356] "Generic (PLEG): container finished" podID="3299c8d5-27d2-4645-900a-837a2f2e42b5" containerID="0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547" exitCode=0
Jul 27 21:10:36.503185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.503013    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"
Jul 27 21:10:36.503185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.503054    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" event={"ID":"3299c8d5-27d2-4645-900a-837a2f2e42b5","Type":"ContainerDied","Data":"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"}
Jul 27 21:10:36.503185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.503109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx" event={"ID":"3299c8d5-27d2-4645-900a-837a2f2e42b5","Type":"ContainerDied","Data":"59ca2931789a512c952ac30b5a8bf34b4cde80d690bc9ce8d44c78c6cecfe252"}
Jul 27 21:10:36.503185 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.503132    2560 scope.go:117] "RemoveContainer" containerID="0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"
Jul 27 21:10:36.511781 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.511766    2560 scope.go:117] "RemoveContainer" containerID="0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"
Jul 27 21:10:36.512047 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:36.512029    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547\": container with ID starting with 0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547 not found: ID does not exist" containerID="0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"
Jul 27 21:10:36.512128 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.512053    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547"} err="failed to get container status \"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547\": rpc error: code = NotFound desc = could not find container \"0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547\": container with ID starting with 0b7b425081def64c7d2ba8243f3e96772dc4c5787d76b91815f23f9ac5176547 not found: ID does not exist"
Jul 27 21:10:36.524466 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.524440    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 21:10:36.527965 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:36.527945    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-66b603-f9855dc69-xnfbx"]
Jul 27 21:10:37.726554 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:37.726509    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3299c8d5-27d2-4645-900a-837a2f2e42b5" path="/var/lib/kubelet/pods/3299c8d5-27d2-4645-900a-837a2f2e42b5/volumes"
Jul 27 21:10:47.005148 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:47.000655    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 21:10:47.005148 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:47.001037    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy" containerID="cri-o://e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4" gracePeriod=30
Jul 27 21:10:52.249244 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.249220    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 21:10:52.421556 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.421775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421565    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4jpm\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.421775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421597    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.421775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.421775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421704    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.421775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421751    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.422036 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.422036 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.421855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.422322    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.422322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.422414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert\") pod \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\" (UID: \"e2f8730d-3dd9-428f-8638-9204c7cc29f1\") "
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.423019    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-credential-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.423041    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-socket\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.423285    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.423399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.423461    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:10:52.424663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.424610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:52.431532 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.425675    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm" (OuterVolumeSpecName: "kube-api-access-s4jpm") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "kube-api-access-s4jpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:10:52.431532 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.425904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:10:52.432333 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.432311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2f8730d-3dd9-428f-8638-9204c7cc29f1" (UID: "e2f8730d-3dd9-428f-8638-9204c7cc29f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:10:52.524463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524416    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-token\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524462    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-podinfo\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524471    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-data\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524479    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istio-envoy\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524463 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524487    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f8730d-3dd9-428f-8638-9204c7cc29f1-istiod-ca-cert\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524756 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524495    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jpm\" (UniqueName: \"kubernetes.io/projected/e2f8730d-3dd9-428f-8638-9204c7cc29f1-kube-api-access-s4jpm\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.524756 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.524505    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f8730d-3dd9-428f-8638-9204c7cc29f1-workload-certs\") on node \"ip-10-0-143-74.ec2.internal\" DevicePath \"\""
Jul 27 21:10:52.589940 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.589906    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerID="e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4" exitCode=0
Jul 27 21:10:52.590113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.589969    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"
Jul 27 21:10:52.590113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.589994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerDied","Data":"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"}
Jul 27 21:10:52.590113 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.590090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn" event={"ID":"e2f8730d-3dd9-428f-8638-9204c7cc29f1","Type":"ContainerDied","Data":"e088988d2031e92103dce0342939008f1817f3aa0d23afd07957326b88f49914"}
Jul 27 21:10:52.590287 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.590121    2560 scope.go:117] "RemoveContainer" containerID="e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"
Jul 27 21:10:52.599286 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.599270    2560 scope.go:117] "RemoveContainer" containerID="a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"
Jul 27 21:10:52.606777 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.606761    2560 scope.go:117] "RemoveContainer" containerID="e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"
Jul 27 21:10:52.607093 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:52.607071    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4\": container with ID starting with e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4 not found: ID does not exist" containerID="e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"
Jul 27 21:10:52.607158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.607104    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4"} err="failed to get container status \"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4\": rpc error: code = NotFound desc = could not find container \"e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4\": container with ID starting with e5feecc765808731a8396f86ca2cfe9d8dc35811992c9c366717092826bc6cd4 not found: ID does not exist"
Jul 27 21:10:52.607158 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.607126    2560 scope.go:117] "RemoveContainer" containerID="a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"
Jul 27 21:10:52.607372 ip-10-0-143-74 kubenswrapper[2560]: E0727 21:10:52.607354    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353\": container with ID starting with a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353 not found: ID does not exist" containerID="a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"
Jul 27 21:10:52.607440 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.607380    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353"} err="failed to get container status \"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353\": rpc error: code = NotFound desc = could not find container \"a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353\": container with ID starting with a8b7bd8f6450fb39d606836146bd3b7112d9b12d299023a4d367ea89db244353 not found: ID does not exist"
Jul 27 21:10:52.613011 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.612992    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 21:10:52.616884 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:52.616862    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-66b603-openshift-default-66c67b9476-92bjn"]
Jul 27 21:10:53.300294 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:53.300264    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbc7b97fc-8xs5p_9cacd101-c89a-4fd0-9749-6e06c4f5eed9/manager/0.log"
Jul 27 21:10:53.681885 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:53.681857    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8fd6b656-22wbs_07bde055-9699-4385-9b10-4974e577e439/copy-manifests/0.log"
Jul 27 21:10:53.692703 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:53.692659    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8fd6b656-22wbs_07bde055-9699-4385-9b10-4974e577e439/manager/0.log"
Jul 27 21:10:53.726662 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:53.726630    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" path="/var/lib/kubelet/pods/e2f8730d-3dd9-428f-8638-9204c7cc29f1/volumes"
Jul 27 21:10:54.389029 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:54.388991    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753070-x2mt4_8b6877e1-462c-4eae-91e1-830527e3c5ad/cleanup/0.log"
Jul 27 21:10:54.500759 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:54.500734    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753085-mgm5c_bbee42f7-106c-48a8-9b59-a00f24921b9b/cleanup/0.log"
Jul 27 21:10:54.605448 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:54.605420    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753100-cz67s_f8756535-e34c-4b60-ad5f-077fb1fd99cf/cleanup/0.log"
Jul 27 21:10:56.552493 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:56.552460    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qrn8r_45e0ccdf-6f35-40cb-a662-2fa1c20cef28/manager/0.log"
Jul 27 21:10:57.454651 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:57.454603    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b675654c-xvfkk_6b0807fb-7a28-4976-8fbf-126a94f42d58/kube-auth-proxy/0.log"
Jul 27 21:10:58.384120 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:58.384091    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-czr7n_24b58f60-b348-4bde-9ae7-b22417484de1/main/0.log"
Jul 27 21:10:58.492797 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:10:58.492771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-m7zwx_1c689fb6-96bd-47f0-afbb-bf062cb2b94a/main/0.log"
Jul 27 21:11:04.092591 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n5ws2/must-gather-4tghl"]
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092935    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092948    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092958    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3299c8d5-27d2-4645-900a-837a2f2e42b5" containerName="maas-api"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092963    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3299c8d5-27d2-4645-900a-837a2f2e42b5" containerName="maas-api"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092969    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092975    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092982    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092988    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.092997    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093003    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093009    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1682669-bd42-4fc3-9faa-5f9043567482" containerName="maas-api"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093014    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1682669-bd42-4fc3-9faa-5f9043567482" containerName="maas-api"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093020    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093027    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093032    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093037    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093049    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093054    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093062    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093067    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093074    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093074 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093079    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093148    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1682669-bd42-4fc3-9faa-5f9043567482" containerName="maas-api"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093156    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093162    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3299c8d5-27d2-4645-900a-837a2f2e42b5" containerName="maas-api"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093168    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093175    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093180    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093186    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f8730d-3dd9-428f-8638-9204c7cc29f1" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093192    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093197    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdc1096-182c-4115-99a1-e49df98662c3" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093202    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="09039a24-aa09-427e-82cb-c3365de91622" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093206    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093211    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="446b3ac9-213f-4494-80c5-5ccbb8e43431" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093268    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.093905 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.093273    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c271fe45-1ab2-4f78-8b87-05232627bc64" containerName="istio-proxy"
Jul 27 21:11:04.096450 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.096429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.100922 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.100898    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n5ws2\"/\"openshift-service-ca.crt\""
Jul 27 21:11:04.101037 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.100923    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n5ws2\"/\"kube-root-ca.crt\""
Jul 27 21:11:04.108440 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.108414    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n5ws2/must-gather-4tghl"]
Jul 27 21:11:04.122847 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.122824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/acef0265-2490-4713-8166-59dd154ddc69-must-gather-output\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.122966 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.122871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5n4\" (UniqueName: \"kubernetes.io/projected/acef0265-2490-4713-8166-59dd154ddc69-kube-api-access-2l5n4\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.224150 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.224119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/acef0265-2490-4713-8166-59dd154ddc69-must-gather-output\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.224309 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.224168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5n4\" (UniqueName: \"kubernetes.io/projected/acef0265-2490-4713-8166-59dd154ddc69-kube-api-access-2l5n4\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.224482 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.224461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/acef0265-2490-4713-8166-59dd154ddc69-must-gather-output\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.232960 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.232933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5n4\" (UniqueName: \"kubernetes.io/projected/acef0265-2490-4713-8166-59dd154ddc69-kube-api-access-2l5n4\") pod \"must-gather-4tghl\" (UID: \"acef0265-2490-4713-8166-59dd154ddc69\") " pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.405641 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.405611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n5ws2/must-gather-4tghl"
Jul 27 21:11:04.536461 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.536433    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n5ws2/must-gather-4tghl"]
Jul 27 21:11:04.538506 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:11:04.538480    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacef0265_2490_4713_8166_59dd154ddc69.slice/crio-53220653fbd1d4528e3fa3f09c2611004a5173dabda17520d3ddea7f5374bdea WatchSource:0}: Error finding container 53220653fbd1d4528e3fa3f09c2611004a5173dabda17520d3ddea7f5374bdea: Status 404 returned error can't find the container with id 53220653fbd1d4528e3fa3f09c2611004a5173dabda17520d3ddea7f5374bdea
Jul 27 21:11:04.540231 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.540215    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:11:04.648059 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:04.648025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n5ws2/must-gather-4tghl" event={"ID":"acef0265-2490-4713-8166-59dd154ddc69","Type":"ContainerStarted","Data":"53220653fbd1d4528e3fa3f09c2611004a5173dabda17520d3ddea7f5374bdea"}
Jul 27 21:11:05.660469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:05.660392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n5ws2/must-gather-4tghl" event={"ID":"acef0265-2490-4713-8166-59dd154ddc69","Type":"ContainerStarted","Data":"86c181d278634a102b6d99b23bf4b33735a8a1d5a5eeffa4c1bc37789cc39049"}
Jul 27 21:11:05.660469 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:05.660448    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n5ws2/must-gather-4tghl" event={"ID":"acef0265-2490-4713-8166-59dd154ddc69","Type":"ContainerStarted","Data":"1ee5b91762d8f5ca4e1d83a12d63c0b4f15263500f8ba03024c6455c81a97e61"}
Jul 27 21:11:05.678518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:05.678469    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n5ws2/must-gather-4tghl" podStartSLOduration=0.787710434 podStartE2EDuration="1.678451735s" totalImagesPullingTime="890.741301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:05.677194681 +0000 UTC m=+3198.526961512" watchObservedRunningTime="2026-07-27 21:11:05.678451735 +0000 UTC m=+3198.528218566"
Jul 27 21:11:07.040782 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:07.040631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-npmtt_3c95815a-2734-4e41-82bf-bd4d0f5883f2/global-pull-secret-syncer/0.log"
Jul 27 21:11:07.130054 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:07.130019    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z5hqg_61fb11b7-fbab-4120-87a5-e4dce9efc13c/konnectivity-agent/0.log"
Jul 27 21:11:07.249461 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:07.249408    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-74.ec2.internal_f84ce0d9219c2a58af44c132cfd94a88/haproxy/0.log"
Jul 27 21:11:12.721404 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:12.720539    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qrn8r_45e0ccdf-6f35-40cb-a662-2fa1c20cef28/manager/0.log"
Jul 27 21:11:14.796885 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:14.796854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qkb8d_c0338909-e8b6-4b62-aa8a-e581a44479dd/kube-state-metrics/0.log"
Jul 27 21:11:14.816062 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:14.816033    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qkb8d_c0338909-e8b6-4b62-aa8a-e581a44479dd/kube-rbac-proxy-main/0.log"
Jul 27 21:11:14.838398 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:14.838371    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-qkb8d_c0338909-e8b6-4b62-aa8a-e581a44479dd/kube-rbac-proxy-self/0.log"
Jul 27 21:11:14.872238 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:14.872199    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b795b9688-fmvnd_90244c4a-8765-4fab-8c5a-b03368dfb9fa/metrics-server/0.log"
Jul 27 21:11:14.899500 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:14.899434    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-dhtz4_a6d08629-b41d-47e4-b1ac-110fcd2862ba/monitoring-plugin/0.log"
Jul 27 21:11:15.102328 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.102249    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfgdx_a1f77da5-3a70-4181-87e5-91b79fa039b4/node-exporter/0.log"
Jul 27 21:11:15.137672 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.137458    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfgdx_a1f77da5-3a70-4181-87e5-91b79fa039b4/kube-rbac-proxy/0.log"
Jul 27 21:11:15.159661 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.159629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfgdx_a1f77da5-3a70-4181-87e5-91b79fa039b4/init-textfile/0.log"
Jul 27 21:11:15.190654 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.190619    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pfnqb_6c03c33a-592b-41ab-8f41-f5aad4f3791a/kube-rbac-proxy-main/0.log"
Jul 27 21:11:15.216325 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.216294    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pfnqb_6c03c33a-592b-41ab-8f41-f5aad4f3791a/kube-rbac-proxy-self/0.log"
Jul 27 21:11:15.239260 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.239214    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pfnqb_6c03c33a-592b-41ab-8f41-f5aad4f3791a/openshift-state-metrics/0.log"
Jul 27 21:11:15.279967 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.279937    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/prometheus/0.log"
Jul 27 21:11:15.304753 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.304725    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/config-reloader/0.log"
Jul 27 21:11:15.326661 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.326632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/thanos-sidecar/0.log"
Jul 27 21:11:15.350336 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.350306    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/kube-rbac-proxy-web/0.log"
Jul 27 21:11:15.381553 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.381527    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/kube-rbac-proxy/0.log"
Jul 27 21:11:15.403775 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.403749    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/kube-rbac-proxy-thanos/0.log"
Jul 27 21:11:15.427546 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.427513    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ff3c5491-7869-4b0e-a8c4-2d5aeec18282/init-config-reloader/0.log"
Jul 27 21:11:15.472243 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.472213    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zgfqw_35733880-6d3d-4469-b635-d23a80352b65/prometheus-operator/0.log"
Jul 27 21:11:15.493090 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.493061    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zgfqw_35733880-6d3d-4469-b635-d23a80352b65/kube-rbac-proxy/0.log"
Jul 27 21:11:15.526607 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.526554    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-frw9p_1a74d76a-4fff-49b3-96f4-f8605dbace98/prometheus-operator-admission-webhook/0.log"
Jul 27 21:11:15.633982 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.633890    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/thanos-query/0.log"
Jul 27 21:11:15.655955 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.655908    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/kube-rbac-proxy-web/0.log"
Jul 27 21:11:15.676959 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.676921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/kube-rbac-proxy/0.log"
Jul 27 21:11:15.700061 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.700036    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/prom-label-proxy/0.log"
Jul 27 21:11:15.721802 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.721773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/kube-rbac-proxy-rules/0.log"
Jul 27 21:11:15.742663 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:15.742636    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84544c7c78-s2pqk_da6ebdc5-e571-41f7-ab58-9afd88a329e6/kube-rbac-proxy-metrics/0.log"
Jul 27 21:11:17.101189 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.101155    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"]
Jul 27 21:11:17.111315 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.111291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.114453 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.114423    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n5ws2\"/\"default-dockercfg-4f785\""
Jul 27 21:11:17.116746 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.116717    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"]
Jul 27 21:11:17.169835 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.169789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qp9\" (UniqueName: \"kubernetes.io/projected/faca82f6-0eb5-4957-8de3-605f019415bb-kube-api-access-c6qp9\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.170005 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.169904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-sys\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.170005 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.169949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-podres\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.170005 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.169976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-lib-modules\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.170005 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.170002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-proc\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.271344 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.271298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-sys\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.271518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.271380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-podres\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.271518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.271410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-lib-modules\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.271518 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.271441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-proc\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.271708 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.271519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qp9\" (UniqueName: \"kubernetes.io/projected/faca82f6-0eb5-4957-8de3-605f019415bb-kube-api-access-c6qp9\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.272070 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.272043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-podres\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.272169 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.272088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-lib-modules\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.272169 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.272115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-sys\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.272169 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.272141    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faca82f6-0eb5-4957-8de3-605f019415bb-proc\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.280863 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.280835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qp9\" (UniqueName: \"kubernetes.io/projected/faca82f6-0eb5-4957-8de3-605f019415bb-kube-api-access-c6qp9\") pod \"perf-node-gather-daemonset-lbf5q\" (UID: \"faca82f6-0eb5-4957-8de3-605f019415bb\") " pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.426759 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.426516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:17.633388 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.633355    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"]
Jul 27 21:11:17.635690 ip-10-0-143-74 kubenswrapper[2560]: W0727 21:11:17.635645    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaca82f6_0eb5_4957_8de3_605f019415bb.slice/crio-6a34b1070766995035401671389258177e26230ccb8dfc52c88bc8fe310e305e WatchSource:0}: Error finding container 6a34b1070766995035401671389258177e26230ccb8dfc52c88bc8fe310e305e: Status 404 returned error can't find the container with id 6a34b1070766995035401671389258177e26230ccb8dfc52c88bc8fe310e305e
Jul 27 21:11:17.749754 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:17.749663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q" event={"ID":"faca82f6-0eb5-4957-8de3-605f019415bb","Type":"ContainerStarted","Data":"6a34b1070766995035401671389258177e26230ccb8dfc52c88bc8fe310e305e"}
Jul 27 21:11:18.195826 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:18.195769    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-rgnjp_0bf431f0-58e7-40a6-8714-5a666b9a2f5a/download-server/0.log"
Jul 27 21:11:18.757744 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:18.757703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q" event={"ID":"faca82f6-0eb5-4957-8de3-605f019415bb","Type":"ContainerStarted","Data":"5aa54f7acb60c6577df5094dc9d5c5bada9e0b561d32c1c6230e1d5440a169d8"}
Jul 27 21:11:18.757944 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:18.757834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:18.774364 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:18.774310    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q" podStartSLOduration=1.774294193 podStartE2EDuration="1.774294193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:11:18.772345135 +0000 UTC m=+3211.622111967" watchObservedRunningTime="2026-07-27 21:11:18.774294193 +0000 UTC m=+3211.624061024"
Jul 27 21:11:19.745056 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:19.745023    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-552n5_f5c1f8bc-723b-4b13-a86c-93c3f0037210/dns/0.log"
Jul 27 21:11:19.766258 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:19.766224    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-552n5_f5c1f8bc-723b-4b13-a86c-93c3f0037210/kube-rbac-proxy/0.log"
Jul 27 21:11:19.904566 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:19.904539    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-crkxs_dba724cb-8bb8-4f73-be21-ab451f82a6ff/dns-node-resolver/0.log"
Jul 27 21:11:20.470970 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:20.470940    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8gdvn_7d905ce2-dab1-4209-afa0-8107f39f9124/node-ca/0.log"
Jul 27 21:11:21.634502 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:21.634473    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b675654c-xvfkk_6b0807fb-7a28-4976-8fbf-126a94f42d58/kube-auth-proxy/0.log"
Jul 27 21:11:22.632837 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:22.632785    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xnms9_32bab317-fce0-4e9c-9f72-af34be8b63e0/serve-healthcheck-canary/0.log"
Jul 27 21:11:23.196567 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:23.196535    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwgnd_012af03a-194e-44ab-9e01-54e6427580d2/kube-rbac-proxy/0.log"
Jul 27 21:11:23.217925 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:23.217895    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwgnd_012af03a-194e-44ab-9e01-54e6427580d2/exporter/0.log"
Jul 27 21:11:23.244080 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:23.244049    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fwgnd_012af03a-194e-44ab-9e01-54e6427580d2/extractor/0.log"
Jul 27 21:11:24.774966 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:24.774936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n5ws2/perf-node-gather-daemonset-lbf5q"
Jul 27 21:11:25.980214 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:25.980127    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cbc7b97fc-8xs5p_9cacd101-c89a-4fd0-9749-6e06c4f5eed9/manager/0.log"
Jul 27 21:11:26.162049 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:26.162011    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8fd6b656-22wbs_07bde055-9699-4385-9b10-4974e577e439/manager/0.log"
Jul 27 21:11:26.185430 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:26.185405    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8fd6b656-22wbs_07bde055-9699-4385-9b10-4974e577e439/copy-manifests/0.log"
Jul 27 21:11:27.679716 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:27.679668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57bc674ff4-bwkqq_bd39830f-b6eb-4c8a-8c1a-57efca5bb4d5/manager/0.log"
Jul 27 21:11:27.763269 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:27.763210    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hjxmh_027b3c70-f1cd-4527-8ad6-32a4f6a6094b/openshift-lws-operator/0.log"
Jul 27 21:11:27.840286 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:27.840255    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b675654c-xvfkk_6b0807fb-7a28-4976-8fbf-126a94f42d58/kube-auth-proxy/0.log"
Jul 27 21:11:29.592402 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:29.592316    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b675654c-xvfkk_6b0807fb-7a28-4976-8fbf-126a94f42d58/kube-auth-proxy/0.log"
Jul 27 21:11:34.322020 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.321988    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:11:34.347151 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.347122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/egress-router-binary-copy/0.log"
Jul 27 21:11:34.368737 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.368712    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/cni-plugins/0.log"
Jul 27 21:11:34.389996 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.389965    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/bond-cni-plugin/0.log"
Jul 27 21:11:34.424228 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.424193    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/routeoverride-cni/0.log"
Jul 27 21:11:34.471109 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.471045    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/whereabouts-cni-bincopy/0.log"
Jul 27 21:11:34.524077 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.524042    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dq6h9_3f2a7e81-83ab-41cc-b66d-c1ae1592ad14/whereabouts-cni/0.log"
Jul 27 21:11:34.625578 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.625514    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-j5qpn_5e7f7396-c4d5-4a5d-86c9-7a2e013fb365/kube-multus/0.log"
Jul 27 21:11:34.705207 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.705152    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gvsbh_1da0167a-bfd5-4791-a03c-52fc3795d410/network-metrics-daemon/0.log"
Jul 27 21:11:34.722876 ip-10-0-143-74 kubenswrapper[2560]: I0727 21:11:34.722784    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gvsbh_1da0167a-bfd5-4791-a03c-52fc3795d410/kube-rbac-proxy/0.log"