Jul 16 21:45:32.844652 ip-10-0-140-81 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:45:32.844663 ip-10-0-140-81 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:45:32.844670 ip-10-0-140-81 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:45:32.844878 ip-10-0-140-81 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:45:42.890457 ip-10-0-140-81 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:45:42.890478 ip-10-0-140-81 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 96a4707c04ab4e65a4e34b32b195c05c --
Jul 16 21:47:21.368731 ip-10-0-140-81 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.783988    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791929    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791945    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791949    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791952    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791955    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791958    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791960    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:21.831299 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791963    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791966    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791969    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791972    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791974    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791977    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791980    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791982    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791985    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791987    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791990    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791993    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791995    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.791998    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792001    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792003    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792006    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792010    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792014    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792017    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.160714 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792020    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:21.865228 ip-10-0-140-81 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792023    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792028    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792032    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792035    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792038    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792041    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792044    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792048    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792050    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792053    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792056    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792058    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792061    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792063    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792066    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792069    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792071    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792074    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792076    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.173330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792079    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792081    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792084    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792088    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792092    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792095    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792098    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792101    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792103    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792106    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792109    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792112    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792114    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792117    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792119    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792122    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792124    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792127    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792129    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.175586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792132    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792134    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792137    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792139    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792142    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792144    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792147    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792149    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792152    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792155    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792157    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792160    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792163    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792165    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792168    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792170    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792173    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792176    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792179    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792182    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.176297 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792184    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792187    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792189    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792192    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792195    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792197    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792200    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792202    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792205    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792207    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792210    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792212    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.792215    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793565    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793573    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793576    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793579    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793583    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793586    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793589    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.177336 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793592    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793595    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793598    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793601    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793605    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793608    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793610    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793613    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793616    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793619    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793623    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793626    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793628    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793631    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793634    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793636    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793639    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793642    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793645    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.178330 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793648    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793651    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793654    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793657    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793660    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793662    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793665    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793667    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793670    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793673    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793675    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793678    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793681    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793684    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793687    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793690    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793692    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793695    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793698    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793700    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.179349 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793703    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793706    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793708    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793712    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793717    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793719    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793722    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793725    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793728    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793731    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793733    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793736    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793739    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793741    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793744    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793746    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793749    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793751    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793754    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793757    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.180322 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793759    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793762    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793766    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793771    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793774    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793777    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793780    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793783    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793786    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793788    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793791    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793794    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793796    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793799    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793801    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793804    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793807    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793810    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793812    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793816    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.181342 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793818    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793821    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793824    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793826    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793829    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793832    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793835    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793837    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793840    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793843    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793847    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793850    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.793853    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794265    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794275    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794279    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794282    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794285    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.182279 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794289    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794292    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794295    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794298    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794301    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794304    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794307    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794310    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794314    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794317    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794319    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794324    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794326    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794329    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794332    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794334    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794337    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794340    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794342    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794345    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.183259 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794347    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794350    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794352    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794355    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794358    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794362    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794365    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794368    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794373    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794376    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794379    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794382    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794384    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794387    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794390    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794392    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794395    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794397    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794400    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794403    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.184866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794405    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794408    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794411    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794413    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794417    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794420    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794422    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794425    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794428    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794431    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794434    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794437    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794439    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794442    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794444    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794448    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794450    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794470    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794473    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.185803 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794477    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794481    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794484    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794487    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794489    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794492    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794494    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794497    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794500    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794502    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794505    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794507    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794510    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794513    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794516    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794518    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794521    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794523    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794527    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794529    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.186782 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794532    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794534    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794537    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794539    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794542    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794544    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794546    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794549    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794551    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794554    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794556    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794559    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794563    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794566    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.794569    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794649    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794656    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794663    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794667    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794671    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794674    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:47:22.187769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794679    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794683    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794687    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794690    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794693    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794697    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794700    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794703    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794706    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794709    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794712    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794715    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794719    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794721    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794724    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794727    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794731    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794734    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794737    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794740    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794743    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794746    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794749    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794754    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:47:22.188758 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794759    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794761    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794774    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794777    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794780    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794783    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794786    2566 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794788    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794793    2566 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794796    2566 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794799    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794802    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794805    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794808    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794812    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794814    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794817    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794820    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794823    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794826    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794829    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794832    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794835    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794838    2566 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794841    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:47:22.189690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794844    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794847    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794850    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794853    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794856    2566 flags.go:64] FLAG: --help="false"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794859    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794864    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794868    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794871    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794874    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794877    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794880    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794883    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794886    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794889    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794892    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794895    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794898    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794901    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794903    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794906    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794909    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794912    2566 flags.go:64] FLAG: --lock-file=""
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794915    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:47:22.190942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794918    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794920    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794926    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794929    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794932    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794934    2566 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794937    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794941    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794943    2566 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794946    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794950    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794953    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794956    2566 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794959    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794965    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794969    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794972    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794975    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794978    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794981    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794987    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794990    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794994    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:47:22.191895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.794997    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795000    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795005    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795008    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795011    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795014    2566 flags.go:64] FLAG: --port="10250"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795018    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795020    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0565640f6dfff93c9"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795024    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795027    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795030    2566 flags.go:64] FLAG: --register-node="true"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795033    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795036    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795040    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795042    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795045    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795052    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795055    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795058    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795061    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795064    2566 flags.go:64] FLAG: --runonce="false"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795067    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795070    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795075    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795077    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:47:22.193058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795082    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795085    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795088    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795091    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795094    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795097    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795100    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795103    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795106    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795109    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795112    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795117    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795120    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795122    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795126    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795129    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795132    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795135    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795138    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795140    2566 flags.go:64] FLAG: --v="2"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795144    2566 flags.go:64] FLAG: --version="false"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795148    2566 flags.go:64] FLAG: --vmodule=""
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795152    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795155    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795284    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.197996 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795288    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795292    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795295    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795297    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795300    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795304    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795306    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795311    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795313    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795316    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795318    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795321    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795323    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795326    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795329    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795331    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795334    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795337    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795339    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795342    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:22.199224 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795344    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795347    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795350    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795353    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795355    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795358    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795360    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795362    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795365    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795368    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795371    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795373    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795376    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795380    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795384    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795387    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795389    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795393    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795396    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:22.207372 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795400    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795402    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795405    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795407    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795410    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795413    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795416    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795420    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795422    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795425    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795428    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795431    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795433    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795436    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795438    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795441    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795443    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795446    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795448    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.208416 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795464    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795467    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795469    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795472    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795475    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795478    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795481    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795484    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795486    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795489    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795491    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795495    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795498    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795502    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795504    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795507    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795510    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795512    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795515    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795517    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.209230 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795520    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795523    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795525    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795528    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795530    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795533    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795535    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795538    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795541    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795544    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795548    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795551    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795554    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795557    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795559    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795562    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795565    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795568    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795571    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.210073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.795573    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.795581    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.802605    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.802620    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.802639    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.802645    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802699    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802703    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802707    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802710    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802713    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802716    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802719    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802722    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.210866 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802725    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802728    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802732    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802734    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802737    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802740    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802743    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802746    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802748    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802754    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802757    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802759    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802762    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802765    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802768    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802771    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802774    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802777    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802780    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802783    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.266485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802785    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802791    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802794    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802796    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802799    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802801    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802804    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802806    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802809    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802811    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802814    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802817    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802820    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802822    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802825    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802828    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802830    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802833    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802836    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802838    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.267464 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802840    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802845    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802847    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802850    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802853    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802855    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802859    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802863    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802866    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802869    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802872    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802875    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802877    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802881    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802884    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802887    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802889    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802893    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802896    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.268732 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802899    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802901    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802904    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802906    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802908    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802911    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802913    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802917    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802921    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802924    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802927    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802929    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802932    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802934    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802938    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802941    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802943    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802946    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802949    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802952    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.269546 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802954    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802957    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802960    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802962    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802965    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802967    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802972    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802974    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802977    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802980    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802983    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.802985    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.802990    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803099    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803104    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:47:22.270845 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803107    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803110    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803112    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803116    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803120    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803124    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803126    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803129    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803132    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803134    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803137    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803139    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803145    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803147    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803150    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803152    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803155    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803157    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803160    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:47:22.271863 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803162    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803165    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803167    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803170    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803174    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803177    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803179    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803183    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803185    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803188    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803190    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803193    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803195    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803198    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803200    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803202    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803205    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803207    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803210    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803213    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:47:22.272681 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803215    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803217    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803220    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803222    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803224    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803228    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803231    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803234    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803236    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803238    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803241    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803245    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803249    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803252    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803254    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803257    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803260    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803263    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803266    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803268    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:47:22.273723 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803271    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803274    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803276    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803279    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803281    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803283    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803286    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803288    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803291    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803293    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803296    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803298    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803301    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803303    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803306    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803308    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803310    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803314    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803317    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803321    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:47:22.274775 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803324    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803327    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803329    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803332    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803335    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803338    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803340    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803343    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803346    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803349    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803351    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803354    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803356    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803359    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803362    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803364    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803367    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:47:22.275578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:21.803369    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.803374    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.803514    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.805653    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.807112    2566 server.go:1019] "Starting client certificate rotation"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.807227    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.807262    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.809385    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.812370    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.832052    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.833971    2566 log.go:25] "Validated CRI v1 image API"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.834711    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:47:22.276353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.837217    2566 fs.go:135] Filesystem UUIDs: map[63d8fb6b-d288-4eab-894a-9ea81ed1b1b3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d71e2c68-92b9-44d1-8d00-e8319820ef26:/dev/nvme0n1p4]
Jul 16 21:47:22.276891 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.837231    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:47:22.276891 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.840528    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.842499    2566 manager.go:216] Machine: {Timestamp:2026-07-16 21:47:21.841080812 +0000 UTC m=+0.350947884 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099881 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b3d8e92c6623318465b094ff04c7c SystemUUID:ec2b3d8e-92c6-6233-1846-5b094ff04c7c BootID:96a4707c-04ab-4e65-a4e3-4b32b195c05c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9e:92:f0:a9:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9e:92:f0:a9:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:c3:92:1f:e4:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.842587    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.842688    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.846182    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.846204    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.846964    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.846975    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.846982    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.847731    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.849094    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852414    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852527    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852553    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852562    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852577    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852586    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852594    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:47:22.277370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852874    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.852888    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.855075    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.857130    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.857158    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858745    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858763    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858769    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858774    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858782    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858789    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858797    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858802    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858811    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858818    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858826    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.858831    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.860081    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.860089    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.863321    2566 server.go:1268] "Started kubelet"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.863381    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.863475    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.863530    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.864207    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.865017    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.869592    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.871887    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.873108    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.873152    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.873350    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-81.ec2.internal.18c2e368d9dbae46  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-81.ec2.internal,UID:ip-10-0-140-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-81.ec2.internal,},FirstTimestamp:2026-07-16 21:47:21.863294534 +0000 UTC m=+0.373161607,LastTimestamp:2026-07-16 21:47:21.863294534 +0000 UTC m=+0.373161607,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-81.ec2.internal,}"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.874942    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.874959    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.875079    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875663    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875663    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875692    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875758    2566 factory.go:55] Registering systemd factory
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875770    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875806    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875816    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875952    2566 factory.go:153] Registering CRI-O factory
Jul 16 21:47:22.278189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.875964    2566 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.876013    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.876037    2566 factory.go:103] Registering Raw factory
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.876051    2566 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.876911    2566 manager.go:349] Starting recovery of all containers
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.876982    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.877078    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.882687    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.887419    2566 manager.go:354] Recovery completed
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.892686    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895345    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895380    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895390    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895821    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895829    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895840    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.895843    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.896948    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.896970    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.896979    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.897172    2566 policy_none.go:49] "None policy: Start"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.897185    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.897195    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.897782    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-81.ec2.internal.18c2e368dbc511d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-81.ec2.internal,UID:ip-10-0-140-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-81.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-81.ec2.internal,},FirstTimestamp:2026-07-16 21:47:21.895367127 +0000 UTC m=+0.405234203,LastTimestamp:2026-07-16 21:47:21.895367127 +0000 UTC m=+0.405234203,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-81.ec2.internal,}"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.898166    2566 policy_none.go:47] "Start"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.902464    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fq7jf"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.913859    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-81.ec2.internal.18c2e368dbc55630  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-81.ec2.internal,UID:ip-10-0-140-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-81.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-81.ec2.internal,},FirstTimestamp:2026-07-16 21:47:21.895384624 +0000 UTC m=+0.405251696,LastTimestamp:2026-07-16 21:47:21.895384624 +0000 UTC m=+0.405251696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-81.ec2.internal,}"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.913965    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fq7jf"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.975315    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989061    2566 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.989098    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989111    2566 server.go:87] "Starting device plugin registration server"
Jul 16 21:47:22.282178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989428    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989439    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989514    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989600    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:21.989609    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.990183    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:21.990224    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.003840    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.003874    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.003891    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.003919    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.006595    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.084177    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-81.ec2.internal\" not found" node="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.090281    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.091227    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.091268    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.091278    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.091299    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.100192    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.100209    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-81.ec2.internal\": node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.104121    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"]
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.104187    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.104932    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.104962    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.104973    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106267    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106416    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106952    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106973    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106983    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.106984    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.107088    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.107100    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.108147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.108167    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.108797    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.108828    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.108843    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.129028    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-81.ec2.internal\" not found" node="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.283533 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.132963    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-81.ec2.internal\" not found" node="ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.134524    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.177355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d12d6b74a3f6fbeca9855716ed7cfc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-81.ec2.internal\" (UID: \"05d12d6b74a3f6fbeca9855716ed7cfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.177379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.177397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.235519    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d12d6b74a3f6fbeca9855716ed7cfc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-81.ec2.internal\" (UID: \"05d12d6b74a3f6fbeca9855716ed7cfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05d12d6b74a3f6fbeca9855716ed7cfc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-81.ec2.internal\" (UID: \"05d12d6b74a3f6fbeca9855716ed7cfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.291362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.277979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72d6949b9c0eb9994bc9cb1593954dc6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal\" (UID: \"72d6949b9c0eb9994bc9cb1593954dc6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.347386 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.336115    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.484635 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.432801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.484635 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.435835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:22.484635 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.436984    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.579625 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.537728    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.645950 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.638220    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.719347 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.718765    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.949413    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:42:21 +0000 UTC" deadline="2028-05-01 13:51:47.342907136 +0000 UTC"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.949497    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15712h4m24.393415345s"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:22.949458    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.949459    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.949692    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:47:22.949892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.949697    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:47:23.020506 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:22.967904    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:23.094444 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.050584    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:23.094444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.075760    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:47:23.094444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.077339    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:47:23.094444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.077370    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:47:23.094444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.077380    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:47:23.196311 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.151404    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:23.251690 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.251500    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-81.ec2.internal\" not found"
Jul 16 21:47:23.280924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.280905    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:23.290515 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:23.290485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d6949b9c0eb9994bc9cb1593954dc6.slice/crio-9fbeac6844f81bf2de9694bec59a639b72bb36aabb65e950e0410937301579f1 WatchSource:0}: Error finding container 9fbeac6844f81bf2de9694bec59a639b72bb36aabb65e950e0410937301579f1: Status 404 returned error can't find the container with id 9fbeac6844f81bf2de9694bec59a639b72bb36aabb65e950e0410937301579f1
Jul 16 21:47:23.290891 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:23.290874    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d12d6b74a3f6fbeca9855716ed7cfc.slice/crio-4ac02ddb4a8fd85f95188a0f0c31eeed2aa78bad30f5c3adf536f16d3d0b8859 WatchSource:0}: Error finding container 4ac02ddb4a8fd85f95188a0f0c31eeed2aa78bad30f5c3adf536f16d3d0b8859: Status 404 returned error can't find the container with id 4ac02ddb4a8fd85f95188a0f0c31eeed2aa78bad30f5c3adf536f16d3d0b8859
Jul 16 21:47:23.298861 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.298845    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:47:23.376561 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.376526    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:23.388694 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.388673    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:23.389753 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.389741    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal"
Jul 16 21:47:23.404627 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.404608    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:47:23.875517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.875493    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:47:23.889005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.888977    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:47:23.913857 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.913831    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fp4pz"
Jul 16 21:47:23.923415 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.923389    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fp4pz"
Jul 16 21:47:23.949762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.949739    2566 apiserver.go:52] "Watching apiserver"
Jul 16 21:47:23.961526 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.961496    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:47:23.962814 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.962778    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-wc2bm","openshift-image-registry/node-ca-qxjdn","openshift-multus/multus-p8z22","openshift-network-operator/iptables-alerter-cdmpc","openshift-ovn-kubernetes/ovnkube-node-kg7b7","kube-system/konnectivity-agent-tcj9h","kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal","openshift-dns/node-resolver-zjf7s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal","openshift-multus/multus-additional-cni-plugins-h22hw","openshift-multus/network-metrics-daemon-qhcbp","openshift-network-diagnostics/network-check-target-x5mxz","kube-system/global-pull-secret-syncer-mj2sj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"]
Jul 16 21:47:23.964251 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.964230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.966212 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.966161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:23.966309 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.966263    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:23.966992 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.966975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:47:23.967423 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.967423 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.967423 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jlqm2\""
Jul 16 21:47:23.967423 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:47:23.967673 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967441    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.968124 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.967993    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:47:23.969200 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.968583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:23.970158 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.970142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.970750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mxf44\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.971254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.971310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47nfj\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.971699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.971819    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.971875    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.972023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.972172    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.972404    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-df6cq\""
Jul 16 21:47:23.972660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.972543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:47:23.973155 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.972794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.973867 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.973849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:23.974535 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.974401    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:47:23.974641 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.974618    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.974763 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.974746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.974833 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.974806    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:47:23.974886 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.974808    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k5bvz\""
Jul 16 21:47:23.976148 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.975488    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:47:23.976148 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.975488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:23.976148 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.975924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:47:23.976333 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976430    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976540    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5dk5s\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cvhpb\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976697    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.976924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976819    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.977205 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.976975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.977979 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.977964    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qj4t8\""
Jul 16 21:47:23.978894 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.978634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:23.978894 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.978728    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:23.979990 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.979967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:23.980089 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:23.980026    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:23.980089 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.980071    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.980206 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.980120    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:47:23.980206 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.980173    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.981596 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.981577    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:23.983987 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.983832    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6bzxw\""
Jul 16 21:47:23.983987 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.983941    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:23.986597 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-systemd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.986772 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.986878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-kubernetes\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.986878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-socket-dir-parent\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.986878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6vs\" (UniqueName: \"kubernetes.io/projected/03047129-e539-4366-b8f3-14791b800d70-kube-api-access-sm6vs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e901dacd-baf6-481d-946f-1609920866a0-hosts-file\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7d5w\" (UniqueName: \"kubernetes.io/projected/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-kube-api-access-x7d5w\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-systemd-units\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.986954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-kubelet-config\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:23.987051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-config\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-lib-modules\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-tuned\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-bin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-sys-fs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-env-overrides\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-socket-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxrg\" (UniqueName: \"kubernetes.io/projected/e901dacd-baf6-481d-946f-1609920866a0-kube-api-access-xrxrg\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ggd\" (UniqueName: \"kubernetes.io/projected/14d92f62-9c92-4ca2-85e0-558fdd5b688c-kube-api-access-c5ggd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987307 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-agent-certs\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysconfig\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987342    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-netns\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-conf-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-ovn\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-log-socket\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-konnectivity-ca\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0923c7aa-6359-41c6-9c34-3681fe950bf5-host-slash\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-netns\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-hostroot\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0923c7aa-6359-41c6-9c34-3681fe950bf5-iptables-alerter-script\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-registration-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-dbus\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.987784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxhg\" (UniqueName: \"kubernetes.io/projected/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-kube-api-access-cnxhg\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-slash\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-script-lib\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-tmp\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-var-lib-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-bin\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-cni-binary-copy\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-etc-kubernetes\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8r4\" (UniqueName: \"kubernetes.io/projected/0923c7aa-6359-41c6-9c34-3681fe950bf5-kube-api-access-mt8r4\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.987977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-os-release\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-kubelet\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-netd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-serviceca\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-binary-copy\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.988558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-systemd\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-cnibin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcmn5\" (UniqueName: \"kubernetes.io/projected/add9f3d6-8903-428e-b59a-626df05881ad-kube-api-access-fcmn5\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-node-log\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-sys\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwfx\" (UniqueName: \"kubernetes.io/projected/f7e5aea8-303a-42c7-9ffa-426342ba1c83-kube-api-access-fmwfx\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e901dacd-baf6-481d-946f-1609920866a0-tmp-dir\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-host\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-system-cni-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovn-node-metrics-cert\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-host\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-kubelet\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-multus-certs\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-device-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:23.989187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-system-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-k8s-cni-cncf-io\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-multus-daemon-config\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cnibin\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-etc-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-modprobe-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-run\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-var-lib-kubelet\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-os-release\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-multus\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:23.989800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:23.988798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-conf\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.008464 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.008405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal" event={"ID":"05d12d6b74a3f6fbeca9855716ed7cfc","Type":"ContainerStarted","Data":"4ac02ddb4a8fd85f95188a0f0c31eeed2aa78bad30f5c3adf536f16d3d0b8859"}
Jul 16 21:47:24.009700 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.009674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal" event={"ID":"72d6949b9c0eb9994bc9cb1593954dc6","Type":"ContainerStarted","Data":"9fbeac6844f81bf2de9694bec59a639b72bb36aabb65e950e0410937301579f1"}
Jul 16 21:47:24.077249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.077215    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:47:24.088989 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.088962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.088993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-conf\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-systemd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-kubernetes\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-socket-dir-parent\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-systemd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6vs\" (UniqueName: \"kubernetes.io/projected/03047129-e539-4366-b8f3-14791b800d70-kube-api-access-sm6vs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e901dacd-baf6-481d-946f-1609920866a0-hosts-file\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-conf\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7d5w\" (UniqueName: \"kubernetes.io/projected/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-kube-api-access-x7d5w\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-kubernetes\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-systemd-units\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-socket-dir-parent\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-systemd-units\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e901dacd-baf6-481d-946f-1609920866a0-hosts-file\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-kubelet-config\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-config\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-lib-modules\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-kubelet-config\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysctl-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-tuned\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.089521 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-bin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-lib-modules\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-sys-fs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-env-overrides\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-socket-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxrg\" (UniqueName: \"kubernetes.io/projected/e901dacd-baf6-481d-946f-1609920866a0-kube-api-access-xrxrg\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkfjp\" (UniqueName: \"kubernetes.io/projected/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-kube-api-access-mkfjp\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ggd\" (UniqueName: \"kubernetes.io/projected/14d92f62-9c92-4ca2-85e0-558fdd5b688c-kube-api-access-c5ggd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-agent-certs\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysconfig\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-netns\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-conf-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-socket-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-ovn\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-log-socket\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-run-ovn\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.089827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-konnectivity-ca\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-config\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-netns\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-multus-conf-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0923c7aa-6359-41c6-9c34-3681fe950bf5-host-slash\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090247    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-netns\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-konnectivity-ca\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-hostroot\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-bin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0923c7aa-6359-41c6-9c34-3681fe950bf5-iptables-alerter-script\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-registration-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-sys-fs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-dbus\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-registration-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-log-socket\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.090996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0923c7aa-6359-41c6-9c34-3681fe950bf5-host-slash\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9aa83795-0672-4fc2-9607-bbcc4434ece5-dbus\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-env-overrides\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-hostroot\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-run-netns\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-sysconfig\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.090433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxhg\" (UniqueName: \"kubernetes.io/projected/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-kube-api-access-cnxhg\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-slash\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-script-lib\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-tmp\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-var-lib-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-bin\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-cni-binary-copy\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.091741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-etc-kubernetes\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8r4\" (UniqueName: \"kubernetes.io/projected/0923c7aa-6359-41c6-9c34-3681fe950bf5-kube-api-access-mt8r4\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-os-release\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-kubelet\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-netd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-serviceca\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-binary-copy\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0923c7aa-6359-41c6-9c34-3681fe950bf5-iptables-alerter-script\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-systemd\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-cnibin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcmn5\" (UniqueName: \"kubernetes.io/projected/add9f3d6-8903-428e-b59a-626df05881ad-kube-api-access-fcmn5\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-node-log\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.092536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-sys\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-os-release\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwfx\" (UniqueName: \"kubernetes.io/projected/f7e5aea8-303a-42c7-9ffa-426342ba1c83-kube-api-access-fmwfx\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-slash\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e901dacd-baf6-481d-946f-1609920866a0-tmp-dir\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-kubelet\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-host\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-system-cni-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-netd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-cni-bin\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovn-node-metrics-cert\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-host\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-kubelet\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-multus-certs\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-device-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-system-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.093328 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-k8s-cni-cncf-io\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-multus-daemon-config\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cnibin\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-etc-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-modprobe-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-run\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-var-lib-kubelet\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-os-release\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-multus\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovnkube-script-lib\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-cni-multus\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-var-lib-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-etc-kubernetes\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-k8s-cni-cncf-io\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.092871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-serviceca\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-node-log\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.091780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.094207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cnibin\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-etc-openvswitch\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-cni-binary-copy\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-sys\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-system-cni-dir\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-host\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.093599    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e901dacd-baf6-481d-946f-1609920866a0-tmp-dir\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.093702    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:24.593637355 +0000 UTC m=+3.103504435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/add9f3d6-8903-428e-b59a-626df05881ad-multus-daemon-config\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-modprobe-d\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-cni-binary-copy\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14d92f62-9c92-4ca2-85e0-558fdd5b688c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-var-lib-kubelet\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-cnibin\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-run\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-systemd\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-run-multus-certs\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.094991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-os-release\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7e5aea8-303a-42c7-9ffa-426342ba1c83-host\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.093974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-host-var-lib-kubelet\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.094010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/03047129-e539-4366-b8f3-14791b800d70-device-dir\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.094026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/add9f3d6-8903-428e-b59a-626df05881ad-system-cni-dir\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.094914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-tmp\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.095078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7e5aea8-303a-42c7-9ffa-426342ba1c83-etc-tuned\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.095821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78785ffe-be06-4f51-a2e1-a92e54ebcfa4-agent-certs\") pod \"konnectivity-agent-tcj9h\" (UID: \"78785ffe-be06-4f51-a2e1-a92e54ebcfa4\") " pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:24.095924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.095829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14d92f62-9c92-4ca2-85e0-558fdd5b688c-ovn-node-metrics-cert\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.100357 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.100336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ggd\" (UniqueName: \"kubernetes.io/projected/14d92f62-9c92-4ca2-85e0-558fdd5b688c-kube-api-access-c5ggd\") pod \"ovnkube-node-kg7b7\" (UID: \"14d92f62-9c92-4ca2-85e0-558fdd5b688c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.100489 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.100334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7d5w\" (UniqueName: \"kubernetes.io/projected/6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea-kube-api-access-x7d5w\") pod \"node-ca-qxjdn\" (UID: \"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea\") " pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.104929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.104867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6vs\" (UniqueName: \"kubernetes.io/projected/03047129-e539-4366-b8f3-14791b800d70-kube-api-access-sm6vs\") pod \"aws-ebs-csi-driver-node-d9rd4\" (UID: \"03047129-e539-4366-b8f3-14791b800d70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.107952 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.107928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxhg\" (UniqueName: \"kubernetes.io/projected/ba5c004b-a67c-44c9-8ae5-6079cf4eea06-kube-api-access-cnxhg\") pod \"multus-additional-cni-plugins-h22hw\" (UID: \"ba5c004b-a67c-44c9-8ae5-6079cf4eea06\") " pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.108205 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.108183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwfx\" (UniqueName: \"kubernetes.io/projected/f7e5aea8-303a-42c7-9ffa-426342ba1c83-kube-api-access-fmwfx\") pod \"tuned-wc2bm\" (UID: \"f7e5aea8-303a-42c7-9ffa-426342ba1c83\") " pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.108608 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.108498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxrg\" (UniqueName: \"kubernetes.io/projected/e901dacd-baf6-481d-946f-1609920866a0-kube-api-access-xrxrg\") pod \"node-resolver-zjf7s\" (UID: \"e901dacd-baf6-481d-946f-1609920866a0\") " pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.109263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.109234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcmn5\" (UniqueName: \"kubernetes.io/projected/add9f3d6-8903-428e-b59a-626df05881ad-kube-api-access-fcmn5\") pod \"multus-p8z22\" (UID: \"add9f3d6-8903-428e-b59a-626df05881ad\") " pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.110183 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.110162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8r4\" (UniqueName: \"kubernetes.io/projected/0923c7aa-6359-41c6-9c34-3681fe950bf5-kube-api-access-mt8r4\") pod \"iptables-alerter-cdmpc\" (UID: \"0923c7aa-6359-41c6-9c34-3681fe950bf5\") " pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.166411 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.166348    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:24.193339 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.193310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfjp\" (UniqueName: \"kubernetes.io/projected/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-kube-api-access-mkfjp\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.193484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.193351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.193484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.193367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:24.193484 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.193467    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:24.193605 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.193535    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:24.693515041 +0000 UTC m=+3.203382107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:24.202136 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.202114    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:24.202136 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.202141    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:24.202329 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.202156    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:24.202329 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.202229    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:24.702211448 +0000 UTC m=+3.212078525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:24.204749 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.204728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfjp\" (UniqueName: \"kubernetes.io/projected/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-kube-api-access-mkfjp\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.279427 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.279396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h22hw"
Jul 16 21:47:24.287577 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.287549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8z22"
Jul 16 21:47:24.296317 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.296298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cdmpc"
Jul 16 21:47:24.301924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.301906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:24.310716 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.310695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:24.316482 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.316450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4"
Jul 16 21:47:24.323115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.323098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjf7s"
Jul 16 21:47:24.329678 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.329650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxjdn"
Jul 16 21:47:24.335128 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.335109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wc2bm"
Jul 16 21:47:24.596361 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.596331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:24.596557 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.596471    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:24.596557 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.596526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:25.596512596 +0000 UTC m=+4.106379656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:24.697254 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.697215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:24.697428 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.697354    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:24.697428 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.697420    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:25.697400578 +0000 UTC m=+4.207267657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:24.798071 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.798032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:24.798242 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.798155    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:24.798242 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.798168    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:24.798242 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.798176    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:24.798242 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:24.798231    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:25.798218964 +0000 UTC m=+4.308086023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:24.815839 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.815801    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd9f3d6_8903_428e_b59a_626df05881ad.slice/crio-4cd261fcaf847de78c96cfb02a5bb2c2b7e3964837c00d390897a19603748eda WatchSource:0}: Error finding container 4cd261fcaf847de78c96cfb02a5bb2c2b7e3964837c00d390897a19603748eda: Status 404 returned error can't find the container with id 4cd261fcaf847de78c96cfb02a5bb2c2b7e3964837c00d390897a19603748eda
Jul 16 21:47:24.817665 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.817573    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6ed8a8_5f2f_4b83_9409_c7ffc432f1ea.slice/crio-5801935ae9e537fb308eefa11a883eb535d7e61ae429d79cd2c4d14a5aca07bc WatchSource:0}: Error finding container 5801935ae9e537fb308eefa11a883eb535d7e61ae429d79cd2c4d14a5aca07bc: Status 404 returned error can't find the container with id 5801935ae9e537fb308eefa11a883eb535d7e61ae429d79cd2c4d14a5aca07bc
Jul 16 21:47:24.818351 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.818242    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e5aea8_303a_42c7_9ffa_426342ba1c83.slice/crio-2c7b0b94a9c4a91b4b1a0809cb5bd97f570db016de8b42a57f5d482338a6af76 WatchSource:0}: Error finding container 2c7b0b94a9c4a91b4b1a0809cb5bd97f570db016de8b42a57f5d482338a6af76: Status 404 returned error can't find the container with id 2c7b0b94a9c4a91b4b1a0809cb5bd97f570db016de8b42a57f5d482338a6af76
Jul 16 21:47:24.821435 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.821413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0923c7aa_6359_41c6_9c34_3681fe950bf5.slice/crio-bc145c6f2bae56b7f35f3a33eb89500b8407b09df7f3779a4034c40331d896db WatchSource:0}: Error finding container bc145c6f2bae56b7f35f3a33eb89500b8407b09df7f3779a4034c40331d896db: Status 404 returned error can't find the container with id bc145c6f2bae56b7f35f3a33eb89500b8407b09df7f3779a4034c40331d896db
Jul 16 21:47:24.822214 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.822113    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5c004b_a67c_44c9_8ae5_6079cf4eea06.slice/crio-86d3d09cca9ae985f99b6ee326bf96739ec58883a3f6561a3b04e35b1955619f WatchSource:0}: Error finding container 86d3d09cca9ae985f99b6ee326bf96739ec58883a3f6561a3b04e35b1955619f: Status 404 returned error can't find the container with id 86d3d09cca9ae985f99b6ee326bf96739ec58883a3f6561a3b04e35b1955619f
Jul 16 21:47:24.824051 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.823653    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78785ffe_be06_4f51_a2e1_a92e54ebcfa4.slice/crio-0a8d0e9631f3cfe2c9537c3ddcb1efa6a4fe8e4fdbf973df93030592cb7689ad WatchSource:0}: Error finding container 0a8d0e9631f3cfe2c9537c3ddcb1efa6a4fe8e4fdbf973df93030592cb7689ad: Status 404 returned error can't find the container with id 0a8d0e9631f3cfe2c9537c3ddcb1efa6a4fe8e4fdbf973df93030592cb7689ad
Jul 16 21:47:24.824718 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.824602    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode901dacd_baf6_481d_946f_1609920866a0.slice/crio-f96a08d7d1672caf0b8f90b8272653f5e855156e3f5562c1f5d2cf57db3a6638 WatchSource:0}: Error finding container f96a08d7d1672caf0b8f90b8272653f5e855156e3f5562c1f5d2cf57db3a6638: Status 404 returned error can't find the container with id f96a08d7d1672caf0b8f90b8272653f5e855156e3f5562c1f5d2cf57db3a6638
Jul 16 21:47:24.825321 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.825223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d92f62_9c92_4ca2_85e0_558fdd5b688c.slice/crio-1b2a6e54dbd7844491bcbbccae3a751d0d6ec1db5b0d758e7e00a0371137aa5a WatchSource:0}: Error finding container 1b2a6e54dbd7844491bcbbccae3a751d0d6ec1db5b0d758e7e00a0371137aa5a: Status 404 returned error can't find the container with id 1b2a6e54dbd7844491bcbbccae3a751d0d6ec1db5b0d758e7e00a0371137aa5a
Jul 16 21:47:24.826570 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:24.826546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03047129_e539_4366_b8f3_14791b800d70.slice/crio-1a7286ce778155d5927efbd7f851c9d505f82e9b35151cd70ada91383738b9da WatchSource:0}: Error finding container 1a7286ce778155d5927efbd7f851c9d505f82e9b35151cd70ada91383738b9da: Status 404 returned error can't find the container with id 1a7286ce778155d5927efbd7f851c9d505f82e9b35151cd70ada91383738b9da
Jul 16 21:47:24.926045 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.925996    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:42:23 +0000 UTC" deadline="2028-04-12 14:49:05.103461674 +0000 UTC"
Jul 16 21:47:24.926045 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:24.926039    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15257h1m40.177426901s"
Jul 16 21:47:25.012046 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.012014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wc2bm" event={"ID":"f7e5aea8-303a-42c7-9ffa-426342ba1c83","Type":"ContainerStarted","Data":"2c7b0b94a9c4a91b4b1a0809cb5bd97f570db016de8b42a57f5d482338a6af76"}
Jul 16 21:47:25.013076 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.013055    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8z22" event={"ID":"add9f3d6-8903-428e-b59a-626df05881ad","Type":"ContainerStarted","Data":"4cd261fcaf847de78c96cfb02a5bb2c2b7e3964837c00d390897a19603748eda"}
Jul 16 21:47:25.014724 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.014700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal" event={"ID":"05d12d6b74a3f6fbeca9855716ed7cfc","Type":"ContainerStarted","Data":"4f61f1b03ffe87ebbfc4d891126315d6771fdb74d085696f25ad11f10dd42775"}
Jul 16 21:47:25.015699 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.015667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"1b2a6e54dbd7844491bcbbccae3a751d0d6ec1db5b0d758e7e00a0371137aa5a"}
Jul 16 21:47:25.016682 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.016662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tcj9h" event={"ID":"78785ffe-be06-4f51-a2e1-a92e54ebcfa4","Type":"ContainerStarted","Data":"0a8d0e9631f3cfe2c9537c3ddcb1efa6a4fe8e4fdbf973df93030592cb7689ad"}
Jul 16 21:47:25.018041 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.018023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerStarted","Data":"86d3d09cca9ae985f99b6ee326bf96739ec58883a3f6561a3b04e35b1955619f"}
Jul 16 21:47:25.019009 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.018992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxjdn" event={"ID":"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea","Type":"ContainerStarted","Data":"5801935ae9e537fb308eefa11a883eb535d7e61ae429d79cd2c4d14a5aca07bc"}
Jul 16 21:47:25.019973 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.019954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4" event={"ID":"03047129-e539-4366-b8f3-14791b800d70","Type":"ContainerStarted","Data":"1a7286ce778155d5927efbd7f851c9d505f82e9b35151cd70ada91383738b9da"}
Jul 16 21:47:25.020865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.020846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjf7s" event={"ID":"e901dacd-baf6-481d-946f-1609920866a0","Type":"ContainerStarted","Data":"f96a08d7d1672caf0b8f90b8272653f5e855156e3f5562c1f5d2cf57db3a6638"}
Jul 16 21:47:25.024214 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.024193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cdmpc" event={"ID":"0923c7aa-6359-41c6-9c34-3681fe950bf5","Type":"ContainerStarted","Data":"bc145c6f2bae56b7f35f3a33eb89500b8407b09df7f3779a4034c40331d896db"}
Jul 16 21:47:25.034750 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.034703    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-81.ec2.internal" podStartSLOduration=2.0346911 podStartE2EDuration="2.0346911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:25.034666763 +0000 UTC m=+3.544533845" watchObservedRunningTime="2026-07-16 21:47:25.0346911 +0000 UTC m=+3.544558176"
Jul 16 21:47:25.207625 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.207548    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:47:25.603830 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.603796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:25.604021 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.603926    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:25.604021 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.603989    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:27.603971514 +0000 UTC m=+6.113838591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:25.704287 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.704255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:25.704481 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.704403    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:25.704481 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.704475    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:27.704440426 +0000 UTC m=+6.214307492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:25.804715 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:25.804679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:25.804895 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.804877    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:25.804959 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.804902    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:25.804959 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.804915    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:25.805061 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:25.804975    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:27.804955869 +0000 UTC m=+6.314822936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:26.005575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:26.005705    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:26.006033    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:26.006118    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:26.006182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:26.006629 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:26.006261    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:27.620624 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:27.620037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:27.620624 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.620193    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:27.620624 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.620259    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.62024217 +0000 UTC m=+10.130109246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:27.721744 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:27.721142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:27.721744 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.721303    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:27.721744 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.721372    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.721353082 +0000 UTC m=+10.231220159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:27.821754 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:27.821714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:27.821926 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.821884    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:27.821926 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.821912    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:27.821926 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.821926    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:27.822092 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:27.821995    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:31.821966274 +0000 UTC m=+10.331833340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:28.005172 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:28.005139    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:28.005364 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:28.005273    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:28.005440 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:28.005407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:28.005625 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:28.005601    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:28.005715 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:28.005632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:28.005769 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:28.005735    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:30.005109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:30.005038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:30.005737 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:30.005175    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:30.005737 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:30.005567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:30.005737 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:30.005657    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:30.005737 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:30.005710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:30.005977 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:30.005776    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:31.654035 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:31.653993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:31.654543 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.654143    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:31.654543 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.654202    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:39.654186458 +0000 UTC m=+18.164053539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:31.755413 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:31.755367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:31.755590 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.755562    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:31.755664 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.755625    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:39.75560625 +0000 UTC m=+18.265473310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:31.856710 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:31.856674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:31.856873 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.856851    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:31.856931 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.856879    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:31.856931 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.856892    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:31.857047 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:31.856957    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:39.856938427 +0000 UTC m=+18.366805490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:32.005385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:32.005525    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:32.005852    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:32.005941    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:32.005996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:32.006101 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:32.006066    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:34.005426 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:34.005389    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:34.005905 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:34.005547    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:34.005905 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:34.005600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:34.005905 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:34.005615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:34.005905 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:34.005681    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:34.005905 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:34.005750    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:36.004862 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:36.004619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:36.005304 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:36.004620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:36.005304 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:36.004970    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:36.005304 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:36.004702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:36.005304 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:36.005078    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:36.005304 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:36.005127    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:38.004319 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:38.004269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:38.004319 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:38.004304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:38.004875 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:38.004269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:38.004875 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:38.004413    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:38.004875 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:38.004546    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:38.004875 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:38.004633    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:39.717963 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:39.717921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:39.718440 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.718056    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:39.718440 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.718117    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:55.718103014 +0000 UTC m=+34.227970087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:39.818745 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:39.818713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:39.818935 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.818863    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:39.818935 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.818932    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:55.818912841 +0000 UTC m=+34.328779901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:39.919426 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:39.919385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:39.919614 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.919552    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:39.919614 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.919577    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:39.919614 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.919592    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:39.919739 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:39.919664    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:55.919646025 +0000 UTC m=+34.429513096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:40.004524 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:40.004486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:40.004524 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:40.004516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:40.004779 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:40.004486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:40.004779 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:40.004623    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:40.004779 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:40.004704    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:40.004914 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:40.004781    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:42.005775 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:42.005744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:42.006236 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:42.005859    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:42.006236 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:42.005875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:42.006236 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:42.005997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:42.006236 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:42.006042    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:42.006236 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:42.006071    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:44.005048 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.004833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:44.005631 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.004833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:44.005631 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:44.005155    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:44.005631 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:44.005264    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:44.005631 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.005338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:44.005631 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:44.005436    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:44.077194 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.077161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4" event={"ID":"03047129-e539-4366-b8f3-14791b800d70","Type":"ContainerStarted","Data":"2bd66d23eec68701969bfc666c8327ad4fe2bf1622cf693d1a66972d13f86a53"}
Jul 16 21:47:44.078482 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.078440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjf7s" event={"ID":"e901dacd-baf6-481d-946f-1609920866a0","Type":"ContainerStarted","Data":"37461de4826baf1683c4fe6b45871e2ade5792507daaaf5e8ed7d51f0648dd15"}
Jul 16 21:47:44.080025 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.079992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wc2bm" event={"ID":"f7e5aea8-303a-42c7-9ffa-426342ba1c83","Type":"ContainerStarted","Data":"10a0c21844924edcbabd7051c4165afd98cc499c0b7d250a485b9106646c0b15"}
Jul 16 21:47:44.081373 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.081348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8z22" event={"ID":"add9f3d6-8903-428e-b59a-626df05881ad","Type":"ContainerStarted","Data":"0400412f761515b0847a7e83f69ffb0c054b2f8d62a35eb1d391806d5ea08801"}
Jul 16 21:47:44.083412 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.083364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"a2b8a687d95ded86b8e8ad7332e3619b3bcbb131996e577fd6446346e6987838"}
Jul 16 21:47:44.083517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.083426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"a676bacb7e649051a577539a232095337610d67c5f76778937fc392864938c3a"}
Jul 16 21:47:44.083517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.083482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"0fdd583640da81ecdaa338fe5e03a6dbb95671f046399e17e54334e1c8314e13"}
Jul 16 21:47:44.084905 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.084869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tcj9h" event={"ID":"78785ffe-be06-4f51-a2e1-a92e54ebcfa4","Type":"ContainerStarted","Data":"2ca691cf675bb9d072f2d6135f1c6dd91e09b284c908409537772468d15c809b"}
Jul 16 21:47:44.086285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.086259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerStarted","Data":"437eac6f4bdb106ed149d5193aad49ceacf2c439fe22750fd70ea70d34864637"}
Jul 16 21:47:44.087503 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.087479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxjdn" event={"ID":"6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea","Type":"ContainerStarted","Data":"d186a8879a69d5cd1e2f62c0628d305dee00269781a1b0fb2f85a482bff7119e"}
Jul 16 21:47:44.088811 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.088789    2566 generic.go:356] "Generic (PLEG): container finished" podID="72d6949b9c0eb9994bc9cb1593954dc6" containerID="cd033b668404b680ac2cd911d5e3eac6c590f1eb564987ac31f1f136c83c7f75" exitCode=0
Jul 16 21:47:44.088875 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.088820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal" event={"ID":"72d6949b9c0eb9994bc9cb1593954dc6","Type":"ContainerDied","Data":"cd033b668404b680ac2cd911d5e3eac6c590f1eb564987ac31f1f136c83c7f75"}
Jul 16 21:47:44.100108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.100064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zjf7s" podStartSLOduration=3.2759290979999998 podStartE2EDuration="22.100049896s" totalImagesPullingTime="18.824120798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:44.099943649 +0000 UTC m=+22.609810732" watchObservedRunningTime="2026-07-16 21:47:44.100049896 +0000 UTC m=+22.609916967"
Jul 16 21:47:44.137918 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.137236    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p8z22" podStartSLOduration=3.26024104 podStartE2EDuration="22.137216511s" totalImagesPullingTime="18.876975471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:44.135881198 +0000 UTC m=+22.645748280" watchObservedRunningTime="2026-07-16 21:47:44.137216511 +0000 UTC m=+22.647083594"
Jul 16 21:47:44.156143 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.156099    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tcj9h" podStartSLOduration=11.297066627 podStartE2EDuration="22.156084906s" totalImagesPullingTime="10.859018279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:44.155775518 +0000 UTC m=+22.665642606" watchObservedRunningTime="2026-07-16 21:47:44.156084906 +0000 UTC m=+22.665951988"
Jul 16 21:47:44.210219 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.210169    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qxjdn" podStartSLOduration=3.379236216 podStartE2EDuration="22.210152589s" totalImagesPullingTime="18.830916373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:44.209767057 +0000 UTC m=+22.719634140" watchObservedRunningTime="2026-07-16 21:47:44.210152589 +0000 UTC m=+22.720019671"
Jul 16 21:47:44.243382 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:44.243330    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wc2bm" podStartSLOduration=3.411169333 podStartE2EDuration="22.243308916s" totalImagesPullingTime="18.832139583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:44.242870508 +0000 UTC m=+22.752737589" watchObservedRunningTime="2026-07-16 21:47:44.243308916 +0000 UTC m=+22.753176000"
Jul 16 21:47:45.092924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.092672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cdmpc" event={"ID":"0923c7aa-6359-41c6-9c34-3681fe950bf5","Type":"ContainerStarted","Data":"a43dec8fa723f96b1038c074e1287721fdc10baf76ee2c06549b4f9eb5fa4952"}
Jul 16 21:47:45.095421 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.095394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"ce48150aa9828e8fe3d81feff8da93d4cbaddc1732375942f697fe6b4ed25949"}
Jul 16 21:47:45.095560 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.095473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"1f7e60ba316b9f8a4b9a1aeeab955ddbac2896e9b3371409798b9d262545f8ae"}
Jul 16 21:47:45.095560 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.095534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"08ad265ff208e169078a3e88f14171a87ee7112437b2d373e2516f935738fcb1"}
Jul 16 21:47:45.097816 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.097754    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="437eac6f4bdb106ed149d5193aad49ceacf2c439fe22750fd70ea70d34864637" exitCode=0
Jul 16 21:47:45.097933 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.097830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"437eac6f4bdb106ed149d5193aad49ceacf2c439fe22750fd70ea70d34864637"}
Jul 16 21:47:45.100291 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.100237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal" event={"ID":"72d6949b9c0eb9994bc9cb1593954dc6","Type":"ContainerStarted","Data":"abc657f71b34506a0569bf0563a09a147d5d8a1a4c2f57d6dcaf2fd270a113eb"}
Jul 16 21:47:45.129696 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.129652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cdmpc" podStartSLOduration=4.301438013 podStartE2EDuration="23.129638367s" totalImagesPullingTime="18.828200354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:45.112563152 +0000 UTC m=+23.622430235" watchObservedRunningTime="2026-07-16 21:47:45.129638367 +0000 UTC m=+23.639505449"
Jul 16 21:47:45.129904 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.129886    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-81.ec2.internal" podStartSLOduration=22.129882127 podStartE2EDuration="22.129882127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:45.129447098 +0000 UTC m=+23.639314180" watchObservedRunningTime="2026-07-16 21:47:45.129882127 +0000 UTC m=+23.639749208"
Jul 16 21:47:45.691065 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.691043    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:47:45.999796 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:45.999669    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:47:45.69106073Z","UUID":"30367811-d03b-4233-9065-34b76041edff","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:47:46.001154 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.001136    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:47:46.001248 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.001170    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:47:46.004852 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.004830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:46.004852 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.004843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:46.005038 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.004841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:46.005038 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:46.004960    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:46.005152 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:46.005083    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:46.005205 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:46.005169    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:46.104116 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.104078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4" event={"ID":"03047129-e539-4366-b8f3-14791b800d70","Type":"ContainerStarted","Data":"053d19ad522e55cf76d6c7673de9a00e5c660a58f05172b619d390823564701f"}
Jul 16 21:47:46.637665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.637638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:46.638218 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:46.638194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:47.110131 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:47.110097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"b863e759a83462d362ef6546bdfba2f051a403ae6868e2cb2144f71c335bc2b0"}
Jul 16 21:47:48.004964 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.004931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:48.005184 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.004932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:48.005184 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:48.005059    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:48.005304 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.004933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:48.005304 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:48.005214    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:48.005424 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:48.005302    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:48.114835 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.114799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4" event={"ID":"03047129-e539-4366-b8f3-14791b800d70","Type":"ContainerStarted","Data":"16842bd7e10969b6bf9bf072cfa8cdb942205dfd4c5f63a733e7678eaeda6f45"}
Jul 16 21:47:48.138438 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.138355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9rd4" podStartSLOduration=4.374230071 podStartE2EDuration="26.138337669s" totalImagesPullingTime="21.764107598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:48.136707854 +0000 UTC m=+26.646574941" watchObservedRunningTime="2026-07-16 21:47:48.138337669 +0000 UTC m=+26.648204750"
Jul 16 21:47:48.561063 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.561027    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:48.561235 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.561208    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 16 21:47:48.561735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:48.561710    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tcj9h"
Jul 16 21:47:50.004690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.004552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:50.005007 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.004552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:50.005007 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:50.004776    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:50.005007 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:50.004866    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:50.005007 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.004552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:50.005007 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:50.004967    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:50.124158 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.124125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" event={"ID":"14d92f62-9c92-4ca2-85e0-558fdd5b688c","Type":"ContainerStarted","Data":"105c94157cb6696970ec594913b2f5eee526669594795269749d2c4d694d2777"}
Jul 16 21:47:50.124446 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.124418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:50.124552 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.124512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:50.143559 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.143534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:50.158290 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:50.158245    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7" podStartSLOduration=9.268166835 podStartE2EDuration="28.158230267s" totalImagesPullingTime="18.890063432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:50.158035742 +0000 UTC m=+28.667902824" watchObservedRunningTime="2026-07-16 21:47:50.158230267 +0000 UTC m=+28.668097349"
Jul 16 21:47:51.128433 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.128394    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="8597450d03ff05218f9c2069844cb4b2313b586d737fae228dda02332ac72c7a" exitCode=0
Jul 16 21:47:51.129241 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.128494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"8597450d03ff05218f9c2069844cb4b2313b586d737fae228dda02332ac72c7a"}
Jul 16 21:47:51.129241 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.128946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:51.146095 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.146064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:47:51.851249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.851207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mj2sj"]
Jul 16 21:47:51.851428 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.851339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:51.851521 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:51.851448    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:51.855580 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.855556    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5mxz"]
Jul 16 21:47:51.855686 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.855656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:51.855753 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:51.855735    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:51.862081 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.862051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhcbp"]
Jul 16 21:47:51.862190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:51.862155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:51.862286 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:51.862260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:53.004322 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:53.004150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:53.004769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:53.004215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:53.004769 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:53.004386    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:53.004769 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:53.004504    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:53.134893 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:53.134813    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="fec52ffa6d3d32fa2bcdf38137dcfc2698361bfb358129dcb93c1bfe2fb0a645" exitCode=0
Jul 16 21:47:53.135040 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:53.134890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"fec52ffa6d3d32fa2bcdf38137dcfc2698361bfb358129dcb93c1bfe2fb0a645"}
Jul 16 21:47:54.004664 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:54.004633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:54.004992 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:54.004754    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:55.004367 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.004339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:55.004547 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.004342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:55.004547 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.004429    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:55.004547 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.004525    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:55.141511 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.141478    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="2b1980af787e3c948161a28a0a7fd296b742d4b603bdf53fec355bafbbb6cb84" exitCode=0
Jul 16 21:47:55.141511 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.141512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"2b1980af787e3c948161a28a0a7fd296b742d4b603bdf53fec355bafbbb6cb84"}
Jul 16 21:47:55.740395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.740365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:55.740583 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.740546    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:55.740649 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.740602    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret podName:9aa83795-0672-4fc2-9607-bbcc4434ece5 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:27.740586005 +0000 UTC m=+66.250453080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret") pod "global-pull-secret-syncer-mj2sj" (UID: "9aa83795-0672-4fc2-9607-bbcc4434ece5") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:55.841398 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.841364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:55.841587 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.841492    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:55.841587 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.841552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs podName:9a4ec513-a38b-4c89-84f8-4992b1d4fcf3 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:27.841533645 +0000 UTC m=+66.351400706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs") pod "network-metrics-daemon-qhcbp" (UID: "9a4ec513-a38b-4c89-84f8-4992b1d4fcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:55.942102 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:55.942074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:55.942248 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.942190    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:55.942248 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.942203    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:55.942248 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.942212    2566 projected.go:196] Error preparing data for projected volume kube-api-access-qxr6b for pod openshift-network-diagnostics/network-check-target-x5mxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:55.942366 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:55.942256    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b podName:3cffa522-301b-4cdc-98ec-e002c8b057e8 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:27.942241942 +0000 UTC m=+66.452109001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qxr6b" (UniqueName: "kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b") pod "network-check-target-x5mxz" (UID: "3cffa522-301b-4cdc-98ec-e002c8b057e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:56.005232 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:56.005166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:56.005382 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:56.005287    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5mxz" podUID="3cffa522-301b-4cdc-98ec-e002c8b057e8"
Jul 16 21:47:57.004447 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.004413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:57.004888 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.004413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:57.004888 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.004560    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhcbp" podUID="9a4ec513-a38b-4c89-84f8-4992b1d4fcf3"
Jul 16 21:47:57.004888 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.004660    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mj2sj" podUID="9aa83795-0672-4fc2-9607-bbcc4434ece5"
Jul 16 21:47:57.346475 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.346388    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-81.ec2.internal" event="NodeReady"
Jul 16 21:47:57.346636 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.346550    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:47:57.416707 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.416667    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:47:57.418788 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.418766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.418788 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.418778    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8bbmz"]
Jul 16 21:47:57.421685 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.421666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.427587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.425820    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:47:57.427587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.425966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:47:57.427587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.426202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:47:57.427587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.426304    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdfjl\""
Jul 16 21:47:57.427802 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.427612    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"]
Jul 16 21:47:57.430542 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.430317    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"]
Jul 16 21:47:57.430748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.430442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.432575 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.432558    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6ddb94bc54-9kx9k"]
Jul 16 21:47:57.432680 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.432653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.434385 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.434369    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-swksv"]
Jul 16 21:47:57.434490 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.434482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.436010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.435992    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"]
Jul 16 21:47:57.436134 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.436117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.437818 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.437797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.437921 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.437867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 21:47:57.437981 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.437945    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.438133 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.438109    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.443121 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.443106    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zzg68\""
Jul 16 21:47:57.450641 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 21:47:57.450733 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 16 21:47:57.450733 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.450903 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.450999 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 21:47:57.450999 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 21:47:57.451107 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.450880    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.451205 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 16 21:47:57.451274 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 21:47:57.451321 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 21:47:57.451764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451748    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5snzq\""
Jul 16 21:47:57.451865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.451865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.451769    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 16 21:47:57.452084 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452066    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.452211 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452128    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 16 21:47:57.452354 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452338    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.452426 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zpvhf\""
Jul 16 21:47:57.452426 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6pf2r\""
Jul 16 21:47:57.452575 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.452575 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452531    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.452575 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452546    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.452693 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 21:47:57.453145 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.452988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 16 21:47:57.453145 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.453037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zlvd6\""
Jul 16 21:47:57.453278 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.453264    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kfp6l\""
Jul 16 21:47:57.453418 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.453382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.459939 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.459920    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:47:57.465269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.465250    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8bbmz"]
Jul 16 21:47:57.479635 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.479614    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 16 21:47:57.481138 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.481107    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"]
Jul 16 21:47:57.481896 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.481872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"]
Jul 16 21:47:57.489913 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.489887    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:47:57.490494 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.490317    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 16 21:47:57.493220 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.491548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6ddb94bc54-9kx9k"]
Jul 16 21:47:57.493220 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.493163    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vdlb"]
Jul 16 21:47:57.495699 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.495675    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 21:47:57.496598 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.496582    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"]
Jul 16 21:47:57.496766 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.496749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.498440 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.498422    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"]
Jul 16 21:47:57.498582 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.498563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"
Jul 16 21:47:57.500314 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.500298    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"]
Jul 16 21:47:57.500528 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.500508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"
Jul 16 21:47:57.501282 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.501266    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:47:57.501636 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.501511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:47:57.501636 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.501534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9tv9m\""
Jul 16 21:47:57.502613 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.502596    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-swksv"]
Jul 16 21:47:57.502698 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.502622    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jkp4f"]
Jul 16 21:47:57.502754 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.502711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.504644 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.504628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.506170 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506154    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.506418 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nfg5l\""
Jul 16 21:47:57.506645 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lwx28\""
Jul 16 21:47:57.506730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"]
Jul 16 21:47:57.506730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506674    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-scbmf\""
Jul 16 21:47:57.506820 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506524    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.506820 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.506928 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.506913    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.507022 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.507003    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.507347 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.507330    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 21:47:57.507431 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.507355    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 21:47:57.508540 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.508524    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.510250 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.510210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbkjh\""
Jul 16 21:47:57.513841 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.513820    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:47:57.514590 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.514569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:47:57.514860 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.514613    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:47:57.514938 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.514878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vdlb"]
Jul 16 21:47:57.528159 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.528138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jkp4f"]
Jul 16 21:47:57.534169 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.534149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"]
Jul 16 21:47:57.534762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.534742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"]
Jul 16 21:47:57.551726 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.551726 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.551884 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.551884 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqv5\" (UniqueName: \"kubernetes.io/projected/50807f4a-c8ab-411d-9519-fb3f9cc88f30-kube-api-access-zgqv5\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.551884 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.551884 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-tmp\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-service-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp956\" (UniqueName: \"kubernetes.io/projected/d1743492-997c-412c-8691-ec0bdfceec02-kube-api-access-bp956\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-trusted-ca\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.551977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-stats-auth\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/603d23ae-3670-4321-b0c3-15e9d77519e2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.552068 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdf46\" (UniqueName: \"kubernetes.io/projected/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-kube-api-access-xdf46\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq5x\" (UniqueName: \"kubernetes.io/projected/e4b0774f-37d5-498e-a1ff-69f5189b5398-kube-api-access-fjq5x\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-snapshots\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3978b80-d899-4fa8-86b0-48885a155b24-serving-cert\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/603d23ae-3670-4321-b0c3-15e9d77519e2-config\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm27q\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmsj\" (UniqueName: \"kubernetes.io/projected/603d23ae-3670-4321-b0c3-15e9d77519e2-kube-api-access-wfmsj\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1743492-997c-412c-8691-ec0bdfceec02-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.552406 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvql8\" (UniqueName: \"kubernetes.io/projected/d3978b80-d899-4fa8-86b0-48885a155b24-kube-api-access-hvql8\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1743492-997c-412c-8691-ec0bdfceec02-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50807f4a-c8ab-411d-9519-fb3f9cc88f30-serving-cert\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-config\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.552845 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.552598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-default-certificate\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.558930 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.558901    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"]
Jul 16 21:47:57.653139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp956\" (UniqueName: \"kubernetes.io/projected/d1743492-997c-412c-8691-ec0bdfceec02-kube-api-access-bp956\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.653139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-trusted-ca\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.653139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.653420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfqk\" (UniqueName: \"kubernetes.io/projected/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-kube-api-access-hsfqk\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.653420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-stats-auth\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.653420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/603d23ae-3670-4321-b0c3-15e9d77519e2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.653420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7bf600-8418-4011-85ea-de76f6ee0ea3-tmp-dir\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.653420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdf46\" (UniqueName: \"kubernetes.io/projected/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-kube-api-access-xdf46\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq5x\" (UniqueName: \"kubernetes.io/projected/e4b0774f-37d5-498e-a1ff-69f5189b5398-kube-api-access-fjq5x\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.653983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-snapshots\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3978b80-d899-4fa8-86b0-48885a155b24-serving-cert\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/603d23ae-3670-4321-b0c3-15e9d77519e2-config\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm27q\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmsj\" (UniqueName: \"kubernetes.io/projected/603d23ae-3670-4321-b0c3-15e9d77519e2-kube-api-access-wfmsj\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9s6\" (UniqueName: \"kubernetes.io/projected/b8f86347-c83c-401d-8fce-287acba8bf94-kube-api-access-pn9s6\") pod \"network-check-source-7bc5b6bf85-xzfmt\" (UID: \"b8f86347-c83c-401d-8fce-287acba8bf94\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1743492-997c-412c-8691-ec0bdfceec02-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.655402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvql8\" (UniqueName: \"kubernetes.io/projected/d3978b80-d899-4fa8-86b0-48885a155b24-kube-api-access-hvql8\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-trusted-ca\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1743492-997c-412c-8691-ec0bdfceec02-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.654907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1743492-997c-412c-8691-ec0bdfceec02-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.653884    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/603d23ae-3670-4321-b0c3-15e9d77519e2-config\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.655129    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.155112236 +0000 UTC m=+36.664979305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfkv\" (UniqueName: \"kubernetes.io/projected/c2fa3599-1965-4a8d-965d-62a32be2d750-kube-api-access-6cfkv\") pod \"volume-data-source-validator-78cf4c6f6-zmcsq\" (UID: \"c2fa3599-1965-4a8d-965d-62a32be2d750\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50807f4a-c8ab-411d-9519-fb3f9cc88f30-serving-cert\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-config\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-default-certificate\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-snapshots\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.656257 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655829    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7bf600-8418-4011-85ea-de76f6ee0ea3-config-volume\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd24b3-f582-4735-a4eb-96c86a6472cf-cert\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.655984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqv5\" (UniqueName: \"kubernetes.io/projected/50807f4a-c8ab-411d-9519-fb3f9cc88f30-kube-api-access-zgqv5\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-tmp\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkzk\" (UniqueName: \"kubernetes.io/projected/2acd24b3-f582-4735-a4eb-96c86a6472cf-kube-api-access-fxkzk\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntt9\" (UniqueName: \"kubernetes.io/projected/8b7bf600-8418-4011-85ea-de76f6ee0ea3-kube-api-access-bntt9\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-service-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.657052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.656856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-service-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.657646 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.657294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.658252 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.657925    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:47:57.658252 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.657943    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:47:57.658252 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.658043    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.15801661 +0000 UTC m=+36.667883682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:47:57.658252 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3978b80-d899-4fa8-86b0-48885a155b24-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.658252 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.658214    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.158197619 +0000 UTC m=+36.668064703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:47:57.658689 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.658284    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:47:57.658689 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1743492-997c-412c-8691-ec0bdfceec02-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.658689 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3978b80-d899-4fa8-86b0-48885a155b24-tmp\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.658689 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.658433    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.158417386 +0000 UTC m=+36.668284466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:47:57.658689 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3978b80-d899-4fa8-86b0-48885a155b24-serving-cert\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.658941 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/603d23ae-3670-4321-b0c3-15e9d77519e2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.658941 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.658941 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.658932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-stats-auth\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.659488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.659433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50807f4a-c8ab-411d-9519-fb3f9cc88f30-serving-cert\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.659488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.659440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50807f4a-c8ab-411d-9519-fb3f9cc88f30-config\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.660564 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.660542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-default-certificate\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.660742 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.660724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.689146 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.689117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq5x\" (UniqueName: \"kubernetes.io/projected/e4b0774f-37d5-498e-a1ff-69f5189b5398-kube-api-access-fjq5x\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:57.689631 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.689610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqv5\" (UniqueName: \"kubernetes.io/projected/50807f4a-c8ab-411d-9519-fb3f9cc88f30-kube-api-access-zgqv5\") pod \"console-operator-565b7b569d-swksv\" (UID: \"50807f4a-c8ab-411d-9519-fb3f9cc88f30\") " pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.689793 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.689775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp956\" (UniqueName: \"kubernetes.io/projected/d1743492-997c-412c-8691-ec0bdfceec02-kube-api-access-bp956\") pod \"kube-storage-version-migrator-operator-7658d9b749-d7q6z\" (UID: \"d1743492-997c-412c-8691-ec0bdfceec02\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.701348 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.701328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.705024 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.705004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdf46\" (UniqueName: \"kubernetes.io/projected/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-kube-api-access-xdf46\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:57.706835 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.706815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm27q\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:57.707371 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.707352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvql8\" (UniqueName: \"kubernetes.io/projected/d3978b80-d899-4fa8-86b0-48885a155b24-kube-api-access-hvql8\") pod \"insights-operator-555d9d7885-8bbmz\" (UID: \"d3978b80-d899-4fa8-86b0-48885a155b24\") " pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.720338 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.720319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmsj\" (UniqueName: \"kubernetes.io/projected/603d23ae-3670-4321-b0c3-15e9d77519e2-kube-api-access-wfmsj\") pod \"service-ca-operator-5bcbc94c78-vjrnl\" (UID: \"603d23ae-3670-4321-b0c3-15e9d77519e2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.739173 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.739142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8bbmz"
Jul 16 21:47:57.745091 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.745070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"
Jul 16 21:47:57.756691 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.756664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9s6\" (UniqueName: \"kubernetes.io/projected/b8f86347-c83c-401d-8fce-287acba8bf94-kube-api-access-pn9s6\") pod \"network-check-source-7bc5b6bf85-xzfmt\" (UID: \"b8f86347-c83c-401d-8fce-287acba8bf94\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"
Jul 16 21:47:57.756783 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.756724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfkv\" (UniqueName: \"kubernetes.io/projected/c2fa3599-1965-4a8d-965d-62a32be2d750-kube-api-access-6cfkv\") pod \"volume-data-source-validator-78cf4c6f6-zmcsq\" (UID: \"c2fa3599-1965-4a8d-965d-62a32be2d750\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"
Jul 16 21:47:57.756783 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.756770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7bf600-8418-4011-85ea-de76f6ee0ea3-config-volume\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.756797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd24b3-f582-4735-a4eb-96c86a6472cf-cert\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkzk\" (UniqueName: \"kubernetes.io/projected/2acd24b3-f582-4735-a4eb-96c86a6472cf-kube-api-access-fxkzk\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntt9\" (UniqueName: \"kubernetes.io/projected/8b7bf600-8418-4011-85ea-de76f6ee0ea3-kube-api-access-bntt9\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfqk\" (UniqueName: \"kubernetes.io/projected/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-kube-api-access-hsfqk\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7bf600-8418-4011-85ea-de76f6ee0ea3-tmp-dir\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.757712    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.757848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7bf600-8418-4011-85ea-de76f6ee0ea3-config-volume\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.758046    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:57.758397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.758344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b7bf600-8418-4011-85ea-de76f6ee0ea3-tmp-dir\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:57.758976 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.758663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.758976 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.758795    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.25876895 +0000 UTC m=+36.768636026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:57.758976 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:57.758839    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.258826685 +0000 UTC m=+36.768693746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:57.763748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.763727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:47:57.764073 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.764055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acd24b3-f582-4735-a4eb-96c86a6472cf-cert\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.769273 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.769248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"
Jul 16 21:47:57.846476 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.846425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9s6\" (UniqueName: \"kubernetes.io/projected/b8f86347-c83c-401d-8fce-287acba8bf94-kube-api-access-pn9s6\") pod \"network-check-source-7bc5b6bf85-xzfmt\" (UID: \"b8f86347-c83c-401d-8fce-287acba8bf94\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"
Jul 16 21:47:57.846955 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.846906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfkv\" (UniqueName: \"kubernetes.io/projected/c2fa3599-1965-4a8d-965d-62a32be2d750-kube-api-access-6cfkv\") pod \"volume-data-source-validator-78cf4c6f6-zmcsq\" (UID: \"c2fa3599-1965-4a8d-965d-62a32be2d750\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"
Jul 16 21:47:57.851180 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.851151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkzk\" (UniqueName: \"kubernetes.io/projected/2acd24b3-f582-4735-a4eb-96c86a6472cf-kube-api-access-fxkzk\") pod \"ingress-canary-jkp4f\" (UID: \"2acd24b3-f582-4735-a4eb-96c86a6472cf\") " pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:57.889424 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.889359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfqk\" (UniqueName: \"kubernetes.io/projected/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-kube-api-access-hsfqk\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:57.921255 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:57.921189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntt9\" (UniqueName: \"kubernetes.io/projected/8b7bf600-8418-4011-85ea-de76f6ee0ea3-kube-api-access-bntt9\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:58.004930 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.004896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:47:58.009988 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.009966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\""
Jul 16 21:47:58.020173 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.019912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl"]
Jul 16 21:47:58.020936 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.020913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8bbmz"]
Jul 16 21:47:58.021615 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.021598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z"]
Jul 16 21:47:58.022247 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:58.022226    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603d23ae_3670_4321_b0c3_15e9d77519e2.slice/crio-93882bbb3731455b286da7f1a3bd2ff14f0490b0a7890d277116009d7910bc03 WatchSource:0}: Error finding container 93882bbb3731455b286da7f1a3bd2ff14f0490b0a7890d277116009d7910bc03: Status 404 returned error can't find the container with id 93882bbb3731455b286da7f1a3bd2ff14f0490b0a7890d277116009d7910bc03
Jul 16 21:47:58.022340 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.022305    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-swksv"]
Jul 16 21:47:58.022928 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:58.022906    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3978b80_d899_4fa8_86b0_48885a155b24.slice/crio-ced12fb529210708efb724d96fe6cb31cacac8c4e3260d398b197692fb7ed167 WatchSource:0}: Error finding container ced12fb529210708efb724d96fe6cb31cacac8c4e3260d398b197692fb7ed167: Status 404 returned error can't find the container with id ced12fb529210708efb724d96fe6cb31cacac8c4e3260d398b197692fb7ed167
Jul 16 21:47:58.023822 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:58.023799    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1743492_997c_412c_8691_ec0bdfceec02.slice/crio-454cc46a3ea2c8e54793f939607b3f0dd905c005d2cf103b0f650316e9246fef WatchSource:0}: Error finding container 454cc46a3ea2c8e54793f939607b3f0dd905c005d2cf103b0f650316e9246fef: Status 404 returned error can't find the container with id 454cc46a3ea2c8e54793f939607b3f0dd905c005d2cf103b0f650316e9246fef
Jul 16 21:47:58.024682 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:58.024555    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50807f4a_c8ab_411d_9519_fb3f9cc88f30.slice/crio-67a35a055c0d97b51593683ffdeaa5aa4d19ff83a69348761533693022d3e62b WatchSource:0}: Error finding container 67a35a055c0d97b51593683ffdeaa5aa4d19ff83a69348761533693022d3e62b: Status 404 returned error can't find the container with id 67a35a055c0d97b51593683ffdeaa5aa4d19ff83a69348761533693022d3e62b
Jul 16 21:47:58.118029 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.117993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"
Jul 16 21:47:58.126007 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.125985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"
Jul 16 21:47:58.142855 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.142831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jkp4f"
Jul 16 21:47:58.149488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.149448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl" event={"ID":"603d23ae-3670-4321-b0c3-15e9d77519e2","Type":"ContainerStarted","Data":"93882bbb3731455b286da7f1a3bd2ff14f0490b0a7890d277116009d7910bc03"}
Jul 16 21:47:58.150797 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.150771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z" event={"ID":"d1743492-997c-412c-8691-ec0bdfceec02","Type":"ContainerStarted","Data":"454cc46a3ea2c8e54793f939607b3f0dd905c005d2cf103b0f650316e9246fef"}
Jul 16 21:47:58.152004 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.151979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8bbmz" event={"ID":"d3978b80-d899-4fa8-86b0-48885a155b24","Type":"ContainerStarted","Data":"ced12fb529210708efb724d96fe6cb31cacac8c4e3260d398b197692fb7ed167"}
Jul 16 21:47:58.153067 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.153046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-swksv" event={"ID":"50807f4a-c8ab-411d-9519-fb3f9cc88f30","Type":"ContainerStarted","Data":"67a35a055c0d97b51593683ffdeaa5aa4d19ff83a69348761533693022d3e62b"}
Jul 16 21:47:58.162889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.162867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:58.162985 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.162947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:58.162985 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.162975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:58.163090 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.162999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:58.163090 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163032    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:47:58.163090 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163083    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.163067043 +0000 UTC m=+37.672934106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163087    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163140    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.163123667 +0000 UTC m=+37.672990727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163092    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163157    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.163148497 +0000 UTC m=+37.673015557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163157    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:47:58.163240 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.163200    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.163185845 +0000 UTC m=+37.673052924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:47:58.263878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.263846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:58.264092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:58.263900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:58.264092 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.263996    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:58.264092 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.264059    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.264040573 +0000 UTC m=+37.773907633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:58.264092 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.264073    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:58.264268 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:58.264127    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:59.264111684 +0000 UTC m=+37.773978764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:59.004682 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.004468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:47:59.004874 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.004472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:47:59.007805 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.007773    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\""
Jul 16 21:47:59.008890 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.008871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:47:59.008990 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.008915    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:47:59.171593 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.171564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:59.171769 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.171630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:47:59.171769 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.171728    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:47:59.171769 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.171732    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.171715064 +0000 UTC m=+39.681582145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:47:59.171890 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.171880    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.171863775 +0000 UTC m=+39.681730841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:47:59.172302 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.172262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:47:59.172419 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.172338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:47:59.172643 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.172530    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:47:59.172643 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.172551    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:47:59.172643 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.172570    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:47:59.172643 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.172608    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.172592125 +0000 UTC m=+39.682459202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:47:59.172643 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.172638    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.172621416 +0000 UTC m=+39.682488476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:47:59.273017 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.272889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:47:59.273017 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.272946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:47:59.273238 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.273054    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:59.273238 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.273118    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:59.273238 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.273125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.273107967 +0000 UTC m=+39.782975051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:47:59.273238 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:47:59.273185    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:01.273167158 +0000 UTC m=+39.783034224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:59.719711 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.719683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt"]
Jul 16 21:47:59.723749 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:59.723707    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f86347_c83c_401d_8fce_287acba8bf94.slice/crio-4d51d47eb3c29e459e18456d8a89c8cfcce9ad4bab1181eb929a1b6e8ad84dc2 WatchSource:0}: Error finding container 4d51d47eb3c29e459e18456d8a89c8cfcce9ad4bab1181eb929a1b6e8ad84dc2: Status 404 returned error can't find the container with id 4d51d47eb3c29e459e18456d8a89c8cfcce9ad4bab1181eb929a1b6e8ad84dc2
Jul 16 21:47:59.732091 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.731993    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq"]
Jul 16 21:47:59.732364 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:47:59.732329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jkp4f"]
Jul 16 21:47:59.738362 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:47:59.738215    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acd24b3_f582_4735_a4eb_96c86a6472cf.slice/crio-65019af53e4078bc08010774810dddcb612742d767c547447015c2a19557fef8 WatchSource:0}: Error finding container 65019af53e4078bc08010774810dddcb612742d767c547447015c2a19557fef8: Status 404 returned error can't find the container with id 65019af53e4078bc08010774810dddcb612742d767c547447015c2a19557fef8
Jul 16 21:48:00.161519 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:00.161347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt" event={"ID":"b8f86347-c83c-401d-8fce-287acba8bf94","Type":"ContainerStarted","Data":"4d51d47eb3c29e459e18456d8a89c8cfcce9ad4bab1181eb929a1b6e8ad84dc2"}
Jul 16 21:48:00.167117 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:00.167091    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="f20e19fb36e0d062f2fa24a5af4a577aeb52c469d5036fc06093834cfefd47cf" exitCode=0
Jul 16 21:48:00.167260 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:00.167161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"f20e19fb36e0d062f2fa24a5af4a577aeb52c469d5036fc06093834cfefd47cf"}
Jul 16 21:48:00.171389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:00.171363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jkp4f" event={"ID":"2acd24b3-f582-4735-a4eb-96c86a6472cf","Type":"ContainerStarted","Data":"65019af53e4078bc08010774810dddcb612742d767c547447015c2a19557fef8"}
Jul 16 21:48:00.173335 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:00.173284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq" event={"ID":"c2fa3599-1965-4a8d-965d-62a32be2d750","Type":"ContainerStarted","Data":"e6c61eef65d63cf656d849f99b41d15b552ab49b258812f6c98d4855e8e356a4"}
Jul 16 21:48:01.189611 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.189572    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba5c004b-a67c-44c9-8ae5-6079cf4eea06" containerID="db4438974f16437e9d95feda4468250b747343bbde01bf1e4d415657cc612bbe" exitCode=0
Jul 16 21:48:01.190076 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.189647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerDied","Data":"db4438974f16437e9d95feda4468250b747343bbde01bf1e4d415657cc612bbe"}
Jul 16 21:48:01.193555 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.193516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:01.193692 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.193569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:01.193692 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.193629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:01.193814 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.193727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:01.193871 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.193844    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:01.193871 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.193857    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:48:01.193963 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.193908    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.193890668 +0000 UTC m=+43.703757741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:48:01.194269 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.194134    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:48:01.194269 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.194157    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:48:01.194269 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.194185    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.194170922 +0000 UTC m=+43.704038000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:48:01.194269 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.194229    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.194210778 +0000 UTC m=+43.704077855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:48:01.194269 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.194248    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.194238005 +0000 UTC m=+43.704105068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:48:01.294339 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.294273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:01.294339 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:01.294334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:01.294645 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.294515    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:01.294645 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.294588    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.294567114 +0000 UTC m=+43.804434187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:01.294853 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.294660    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:01.294853 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:01.294705    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:05.294693662 +0000 UTC m=+43.804560726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:05.232552 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.232507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.232579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.232613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.232669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232672    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232698    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232712    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232763    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.232742592 +0000 UTC m=+51.742609662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232784    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.232773693 +0000 UTC m=+51.742640758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232799    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.232791979 +0000 UTC m=+51.742659044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232809    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:48:05.233031 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.232858    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.23284396 +0000 UTC m=+51.742711023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:48:05.333208 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.333172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:05.333356 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.333325    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:05.333356 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:05.333348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:05.333422 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.333386    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.333368783 +0000 UTC m=+51.843235847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:05.333475 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.333429    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:05.333507 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:05.333494    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:13.333481177 +0000 UTC m=+51.843348252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:07.208944 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.208906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq" event={"ID":"c2fa3599-1965-4a8d-965d-62a32be2d750","Type":"ContainerStarted","Data":"cccaa34613fbb4b323e6311dc375180d45d2ac51970538088c6f5b17cb0ca90a"}
Jul 16 21:48:07.211005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.210957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8bbmz" event={"ID":"d3978b80-d899-4fa8-86b0-48885a155b24","Type":"ContainerStarted","Data":"93f1173ad3e57230970349ba5dea04041a9505f3e992454419a3df256f35b9a4"}
Jul 16 21:48:07.214887 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.214846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h22hw" event={"ID":"ba5c004b-a67c-44c9-8ae5-6079cf4eea06","Type":"ContainerStarted","Data":"d4958b34e28dceeaadc59dab130c1b9c99a8e3d2dbdea75cc9b9ba8482a072b8"}
Jul 16 21:48:07.291476 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.288228    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zmcsq" podStartSLOduration=3.185319474 podStartE2EDuration="10.288208444s" totalImagesPullingTime="7.10288897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:07.240291765 +0000 UTC m=+45.750158849" watchObservedRunningTime="2026-07-16 21:48:07.288208444 +0000 UTC m=+45.798075526"
Jul 16 21:48:07.291476 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.289188    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h22hw" podStartSLOduration=14.709711500000001 podStartE2EDuration="45.28917693s" totalImagesPullingTime="30.57946543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:07.286909504 +0000 UTC m=+45.796776588" watchObservedRunningTime="2026-07-16 21:48:07.28917693 +0000 UTC m=+45.799044013"
Jul 16 21:48:07.334016 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:07.333563    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-8bbmz" podStartSLOduration=1.490058712 podStartE2EDuration="10.333540233s" totalImagesPullingTime="8.843481521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:07.332314527 +0000 UTC m=+45.842181610" watchObservedRunningTime="2026-07-16 21:48:07.333540233 +0000 UTC m=+45.843407320"
Jul 16 21:48:08.221682 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.221642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z" event={"ID":"d1743492-997c-412c-8691-ec0bdfceec02","Type":"ContainerStarted","Data":"b7a1c581b7afc3e32161c9428d5e9b627be127fede64b82fc1e87f68063e9169"}
Jul 16 21:48:08.223553 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.223525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt" event={"ID":"b8f86347-c83c-401d-8fce-287acba8bf94","Type":"ContainerStarted","Data":"f8fd1a9fca824ee1afce7505b280e3f06f7b3f31a5db4c70b24d2bc8e977054e"}
Jul 16 21:48:08.225158 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.225136    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/0.log"
Jul 16 21:48:08.225276 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.225173    2566 generic.go:356] "Generic (PLEG): container finished" podID="50807f4a-c8ab-411d-9519-fb3f9cc88f30" containerID="5770e6e7caf7a0098977e7b6d33a1780cc608f0fb2561d5eb9b16ec67c0ad29d" exitCode=255
Jul 16 21:48:08.225276 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.225246    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-swksv" event={"ID":"50807f4a-c8ab-411d-9519-fb3f9cc88f30","Type":"ContainerDied","Data":"5770e6e7caf7a0098977e7b6d33a1780cc608f0fb2561d5eb9b16ec67c0ad29d"}
Jul 16 21:48:08.225481 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.225447    2566 scope.go:117] "RemoveContainer" containerID="5770e6e7caf7a0098977e7b6d33a1780cc608f0fb2561d5eb9b16ec67c0ad29d"
Jul 16 21:48:08.227020 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.226976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jkp4f" event={"ID":"2acd24b3-f582-4735-a4eb-96c86a6472cf","Type":"ContainerStarted","Data":"706f1e934a74b5aa11e43e1a5cd4ffa48a1a55e93abb88b1ce5641356528b8f2"}
Jul 16 21:48:08.228503 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.228483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl" event={"ID":"603d23ae-3670-4321-b0c3-15e9d77519e2","Type":"ContainerStarted","Data":"05cd9467702769638fca73fa20c6c5eab7bb94b62bb8b206c851bd15d7d372b7"}
Jul 16 21:48:08.244711 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.244662    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z" podStartSLOduration=2.401303813 podStartE2EDuration="11.244643575s" totalImagesPullingTime="8.843339762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:08.243410578 +0000 UTC m=+46.753277661" watchObservedRunningTime="2026-07-16 21:48:08.244643575 +0000 UTC m=+46.754510659"
Jul 16 21:48:08.289788 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.289729    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-xzfmt" podStartSLOduration=3.871040564 podStartE2EDuration="11.289707391s" totalImagesPullingTime="7.418666827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:08.287812302 +0000 UTC m=+46.797679389" watchObservedRunningTime="2026-07-16 21:48:08.289707391 +0000 UTC m=+46.799574474"
Jul 16 21:48:08.313904 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.313857    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl" podStartSLOduration=2.469711312 podStartE2EDuration="11.313841648s" totalImagesPullingTime="8.844130336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:08.313136059 +0000 UTC m=+46.823003142" watchObservedRunningTime="2026-07-16 21:48:08.313841648 +0000 UTC m=+46.823708729"
Jul 16 21:48:08.338237 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.337911    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jkp4f" podStartSLOduration=4.201788976 podStartE2EDuration="11.337890441s" totalImagesPullingTime="7.136101465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:08.337144292 +0000 UTC m=+46.847011374" watchObservedRunningTime="2026-07-16 21:48:08.337890441 +0000 UTC m=+46.847757525"
Jul 16 21:48:08.608745 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.608711    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6849m"]
Jul 16 21:48:08.621048 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.621021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.626201 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.626169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:48:08.626542 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.626519    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:48:08.626691 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.626580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bcp4l\""
Jul 16 21:48:08.634512 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.634488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6849m"]
Jul 16 21:48:08.666762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.666729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c1ef0ce-398e-4a2a-99f0-071806905ba4-data-volume\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.666899 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.666776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcp4\" (UniqueName: \"kubernetes.io/projected/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-api-access-grcp4\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.666899 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.666853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c1ef0ce-398e-4a2a-99f0-071806905ba4-crio-socket\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.666974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.666926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.666974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.666956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768063 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768063 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768261 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c1ef0ce-398e-4a2a-99f0-071806905ba4-data-volume\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768261 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcp4\" (UniqueName: \"kubernetes.io/projected/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-api-access-grcp4\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768261 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:08.768175    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:08.768261 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:08.768256    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.268232224 +0000 UTC m=+47.778099298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:08.768490 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:08.768270    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:08.768490 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c1ef0ce-398e-4a2a-99f0-071806905ba4-crio-socket\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768490 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:08.768356    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:09.268333295 +0000 UTC m=+47.778200361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:08.768637 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c1ef0ce-398e-4a2a-99f0-071806905ba4-crio-socket\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.768637 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.768558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c1ef0ce-398e-4a2a-99f0-071806905ba4-data-volume\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:08.786750 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:08.786730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcp4\" (UniqueName: \"kubernetes.io/projected/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-api-access-grcp4\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:09.233614 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.233585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:48:09.233992 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.233976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/0.log"
Jul 16 21:48:09.234033 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.234009    2566 generic.go:356] "Generic (PLEG): container finished" podID="50807f4a-c8ab-411d-9519-fb3f9cc88f30" containerID="ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027" exitCode=255
Jul 16 21:48:09.234133 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.234111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-swksv" event={"ID":"50807f4a-c8ab-411d-9519-fb3f9cc88f30","Type":"ContainerDied","Data":"ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027"}
Jul 16 21:48:09.234179 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.234166    2566 scope.go:117] "RemoveContainer" containerID="5770e6e7caf7a0098977e7b6d33a1780cc608f0fb2561d5eb9b16ec67c0ad29d"
Jul 16 21:48:09.234304 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.234291    2566 scope.go:117] "RemoveContainer" containerID="ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027"
Jul 16 21:48:09.234529 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:09.234505    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-swksv_openshift-console-operator(50807f4a-c8ab-411d-9519-fb3f9cc88f30)\"" pod="openshift-console-operator/console-operator-565b7b569d-swksv" podUID="50807f4a-c8ab-411d-9519-fb3f9cc88f30"
Jul 16 21:48:09.273005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.272974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:09.273184 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:09.273021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:09.273184 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:09.273131    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:09.273303 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:09.273221    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.273196479 +0000 UTC m=+48.783063606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:09.273499 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:09.273473    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:09.273572 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:09.273539    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:10.273523103 +0000 UTC m=+48.783390167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.239601 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:10.239575    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:48:10.239978 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:10.239928    2566 scope.go:117] "RemoveContainer" containerID="ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027"
Jul 16 21:48:10.240086 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:10.240067    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-swksv_openshift-console-operator(50807f4a-c8ab-411d-9519-fb3f9cc88f30)\"" pod="openshift-console-operator/console-operator-565b7b569d-swksv" podUID="50807f4a-c8ab-411d-9519-fb3f9cc88f30"
Jul 16 21:48:10.286100 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:10.286061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:10.286100 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:10.286103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:10.286326 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:10.286180    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:10.286326 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:10.286244    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.286228237 +0000 UTC m=+50.796095299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:10.286326 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:10.286244    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.286326 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:10.286307    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:12.28629057 +0000 UTC m=+50.796157630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:10.793955 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:10.793927    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjf7s_e901dacd-baf6-481d-946f-1609920866a0/dns-node-resolver/0.log"
Jul 16 21:48:11.787282 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:11.787254    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxjdn_6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea/node-ca/0.log"
Jul 16 21:48:12.302136 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:12.302100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:12.302136 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:12.302139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:12.302400 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:12.302228    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:48:12.302400 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:12.302297    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.302281396 +0000 UTC m=+54.812148459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : configmap "kube-rbac-proxy" not found
Jul 16 21:48:12.302400 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:12.302345    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:12.302552 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:12.302412    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:16.302395246 +0000 UTC m=+54.812262307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:12.386725 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:12.386699    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jkp4f_2acd24b3-f582-4735-a4eb-96c86a6472cf/serve-healthcheck-canary/0.log"
Jul 16 21:48:13.311347 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.311316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.311355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.311383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.311418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311495    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311510    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311509    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311531    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c74d96db6-gzjmx: secret "image-registry-tls" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.311538455 +0000 UTC m=+67.821405514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : secret "router-metrics-certs-default" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle podName:e4b0774f-37d5-498e-a1ff-69f5189b5398 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.311561389 +0000 UTC m=+67.821428450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle") pod "router-default-6ddb94bc54-9kx9k" (UID: "e4b0774f-37d5-498e-a1ff-69f5189b5398") : configmap references non-existent config key: service-ca.crt
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311584    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls podName:b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.311573164 +0000 UTC m=+67.821440224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-ss5wb" (UID: "b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2") : secret "samples-operator-tls" not found
Jul 16 21:48:13.311801 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.311606    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls podName:e3e9fbb1-79cb-4db4-972d-2c6f35778a1d nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.311589962 +0000 UTC m=+67.821457021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls") pod "image-registry-7c74d96db6-gzjmx" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d") : secret "image-registry-tls" not found
Jul 16 21:48:13.412685 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.412657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:13.412838 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:13.412694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:13.412838 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.412826    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:13.412907 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.412858    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:13.412907 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.412878    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls podName:8b7bf600-8418-4011-85ea-de76f6ee0ea3 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.412865275 +0000 UTC m=+67.922732335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls") pod "dns-default-6vdlb" (UID: "8b7bf600-8418-4011-85ea-de76f6ee0ea3") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:13.412907 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:13.412896    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls podName:9b2c8f8c-0c6f-4891-b65b-b7d004450d38 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:29.412885331 +0000 UTC m=+67.922752394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-lx69d" (UID: "9b2c8f8c-0c6f-4891-b65b-b7d004450d38") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:48:16.336196 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:16.336162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:16.336196 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:16.336194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:16.336688 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:16.336289    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:16.336688 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:16.336341    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls podName:6c1ef0ce-398e-4a2a-99f0-071806905ba4 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:24.336328066 +0000 UTC m=+62.846195127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6849m" (UID: "6c1ef0ce-398e-4a2a-99f0-071806905ba4") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:48:16.336763 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:16.336721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c1ef0ce-398e-4a2a-99f0-071806905ba4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:17.764201 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:17.764167    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:48:17.764628 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:17.764252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:48:17.764668 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:17.764660    2566 scope.go:117] "RemoveContainer" containerID="ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027"
Jul 16 21:48:17.764821 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:48:17.764800    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-swksv_openshift-console-operator(50807f4a-c8ab-411d-9519-fb3f9cc88f30)\"" pod="openshift-console-operator/console-operator-565b7b569d-swksv" podUID="50807f4a-c8ab-411d-9519-fb3f9cc88f30"
Jul 16 21:48:23.150598 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:23.150570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg7b7"
Jul 16 21:48:24.406084 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:24.406043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:24.408344 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:24.408312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c1ef0ce-398e-4a2a-99f0-071806905ba4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6849m\" (UID: \"6c1ef0ce-398e-4a2a-99f0-071806905ba4\") " pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:24.533815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:24.533782    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bcp4l\""
Jul 16 21:48:24.541351 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:24.541328    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6849m"
Jul 16 21:48:24.675480 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:24.675394    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6849m"]
Jul 16 21:48:24.678241 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:24.678216    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1ef0ce_398e_4a2a_99f0_071806905ba4.slice/crio-083bc465bc096286cd23fa5a108a733da73a17364b62e83f006e78c3be86865e WatchSource:0}: Error finding container 083bc465bc096286cd23fa5a108a733da73a17364b62e83f006e78c3be86865e: Status 404 returned error can't find the container with id 083bc465bc096286cd23fa5a108a733da73a17364b62e83f006e78c3be86865e
Jul 16 21:48:25.289635 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:25.289594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6849m" event={"ID":"6c1ef0ce-398e-4a2a-99f0-071806905ba4","Type":"ContainerStarted","Data":"083bc465bc096286cd23fa5a108a733da73a17364b62e83f006e78c3be86865e"}
Jul 16 21:48:27.839119 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.839083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:48:27.842913 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.842889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:48:27.852362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.852341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9aa83795-0672-4fc2-9607-bbcc4434ece5-original-pull-secret\") pod \"global-pull-secret-syncer-mj2sj\" (UID: \"9aa83795-0672-4fc2-9607-bbcc4434ece5\") " pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:48:27.939984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.939934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:48:27.943017 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.942991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:48:27.953474 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:27.953437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ec513-a38b-4c89-84f8-4992b1d4fcf3-metrics-certs\") pod \"network-metrics-daemon-qhcbp\" (UID: \"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3\") " pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:48:28.041308 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.041269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:48:28.044410 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.044384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxr6b\" (UniqueName: \"kubernetes.io/projected/3cffa522-301b-4cdc-98ec-e002c8b057e8-kube-api-access-qxr6b\") pod \"network-check-target-x5mxz\" (UID: \"3cffa522-301b-4cdc-98ec-e002c8b057e8\") " pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:48:28.116739 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.116656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mj2sj"
Jul 16 21:48:28.125109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.125071    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6g595\""
Jul 16 21:48:28.131908 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.131877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhcbp"
Jul 16 21:48:28.318342 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.318302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mc6l9\""
Jul 16 21:48:28.326637 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:28.326598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:48:29.353935 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.353899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:29.354395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.353947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:29.354395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.354128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:29.354395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.354183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:29.354931 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.354907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0774f-37d5-498e-a1ff-69f5189b5398-service-ca-bundle\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:29.356920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.356896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4b0774f-37d5-498e-a1ff-69f5189b5398-metrics-certs\") pod \"router-default-6ddb94bc54-9kx9k\" (UID: \"e4b0774f-37d5-498e-a1ff-69f5189b5398\") " pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:29.357004 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.356919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-ss5wb\" (UID: \"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:29.357004 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.356896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"image-registry-7c74d96db6-gzjmx\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") " pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:29.454810 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.454767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:29.454989 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.454819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:29.457532 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.457502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b7bf600-8418-4011-85ea-de76f6ee0ea3-metrics-tls\") pod \"dns-default-6vdlb\" (UID: \"8b7bf600-8418-4011-85ea-de76f6ee0ea3\") " pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:29.457726 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.457702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b2c8f8c-0c6f-4891-b65b-b7d004450d38-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-lx69d\" (UID: \"9b2c8f8c-0c6f-4891-b65b-b7d004450d38\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:29.536487 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.536432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdfjl\""
Jul 16 21:48:29.543840 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.543810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:29.553314 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.553260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5snzq\""
Jul 16 21:48:29.560852 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.560819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"
Jul 16 21:48:29.561109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.561091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zpvhf\""
Jul 16 21:48:29.569364 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.569332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:29.612693 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.612612    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9tv9m\""
Jul 16 21:48:29.620177 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.620120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:29.643591 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.643560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-scbmf\""
Jul 16 21:48:29.655220 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.654606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"
Jul 16 21:48:29.931040 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:29.931005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d"]
Jul 16 21:48:29.934050 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:29.934022    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2c8f8c_0c6f_4891_b65b_b7d004450d38.slice/crio-8648f67f5e35c96fe41b79d842d7953360736a8b21775d14b710ec19ad08a742 WatchSource:0}: Error finding container 8648f67f5e35c96fe41b79d842d7953360736a8b21775d14b710ec19ad08a742: Status 404 returned error can't find the container with id 8648f67f5e35c96fe41b79d842d7953360736a8b21775d14b710ec19ad08a742
Jul 16 21:48:30.005597 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.005560    2566 scope.go:117] "RemoveContainer" containerID="ceb69ebfa957ae953709fa70acde626d1764d12e1762ef5ca9849429b5d25027"
Jul 16 21:48:30.038979 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.038949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mj2sj"]
Jul 16 21:48:30.044690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.044620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6ddb94bc54-9kx9k"]
Jul 16 21:48:30.047169 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.047051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5mxz"]
Jul 16 21:48:30.047912 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:30.047890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b0774f_37d5_498e_a1ff_69f5189b5398.slice/crio-71e32910c03b5ab9dd51a7c1b7da260bc8b4431907e35f53f6433b7cedbbde97 WatchSource:0}: Error finding container 71e32910c03b5ab9dd51a7c1b7da260bc8b4431907e35f53f6433b7cedbbde97: Status 404 returned error can't find the container with id 71e32910c03b5ab9dd51a7c1b7da260bc8b4431907e35f53f6433b7cedbbde97
Jul 16 21:48:30.049311 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.049285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:48:30.050532 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.050515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhcbp"]
Jul 16 21:48:30.051309 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:30.051157    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e9fbb1_79cb_4db4_972d_2c6f35778a1d.slice/crio-0a42e0cc8fb0882843279285dc3eb476509c71840d57dcf54244c2a660470a92 WatchSource:0}: Error finding container 0a42e0cc8fb0882843279285dc3eb476509c71840d57dcf54244c2a660470a92: Status 404 returned error can't find the container with id 0a42e0cc8fb0882843279285dc3eb476509c71840d57dcf54244c2a660470a92
Jul 16 21:48:30.052263 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:30.052231    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4ec513_a38b_4c89_84f8_4992b1d4fcf3.slice/crio-42e2e2ff816f4445ca54c108efe643077550c9da03d8e118b7c8dedbea9dc3b8 WatchSource:0}: Error finding container 42e2e2ff816f4445ca54c108efe643077550c9da03d8e118b7c8dedbea9dc3b8: Status 404 returned error can't find the container with id 42e2e2ff816f4445ca54c108efe643077550c9da03d8e118b7c8dedbea9dc3b8
Jul 16 21:48:30.112958 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.112903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vdlb"]
Jul 16 21:48:30.122302 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.119886    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb"]
Jul 16 21:48:30.124045 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:30.123983    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7bf600_8418_4011_85ea_de76f6ee0ea3.slice/crio-69ffa648d9027d43f7f2e3fce7aa22dcaec3145ad0af2e994ad4254ee277f8f2 WatchSource:0}: Error finding container 69ffa648d9027d43f7f2e3fce7aa22dcaec3145ad0af2e994ad4254ee277f8f2: Status 404 returned error can't find the container with id 69ffa648d9027d43f7f2e3fce7aa22dcaec3145ad0af2e994ad4254ee277f8f2
Jul 16 21:48:30.309295 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.309258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d" event={"ID":"9b2c8f8c-0c6f-4891-b65b-b7d004450d38","Type":"ContainerStarted","Data":"8648f67f5e35c96fe41b79d842d7953360736a8b21775d14b710ec19ad08a742"}
Jul 16 21:48:30.310424 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.310395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mj2sj" event={"ID":"9aa83795-0672-4fc2-9607-bbcc4434ece5","Type":"ContainerStarted","Data":"37aeebc8f5c5431b5c7c1e8b5754cc653d63110d15733d9b3559564613776518"}
Jul 16 21:48:30.312067 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.312043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5mxz" event={"ID":"3cffa522-301b-4cdc-98ec-e002c8b057e8","Type":"ContainerStarted","Data":"cc6d193d256514e4dd49849438a7e7c6516f07d42d5964a1c4e5e7870298b8c9"}
Jul 16 21:48:30.312165 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.312078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5mxz" event={"ID":"3cffa522-301b-4cdc-98ec-e002c8b057e8","Type":"ContainerStarted","Data":"123a43efd7b776898491d6ae511257e46e50f655ebe725fbeeb4f7a7dc8c98b8"}
Jul 16 21:48:30.312235 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.312206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:48:30.314116 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.314094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:48:30.314214 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.314189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-swksv" event={"ID":"50807f4a-c8ab-411d-9519-fb3f9cc88f30","Type":"ContainerStarted","Data":"597db1e74a4cc4aa5aa7dc45ca70899cec7a7ca492cde1a42e8f036ab5a3626e"}
Jul 16 21:48:30.314537 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.314493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:48:30.316023 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.315959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k" event={"ID":"e4b0774f-37d5-498e-a1ff-69f5189b5398","Type":"ContainerStarted","Data":"a94d01aa03b4757377170806e9b09c4e268ecaafc60626a95f2ec9326f6fd1d6"}
Jul 16 21:48:30.316023 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.315998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k" event={"ID":"e4b0774f-37d5-498e-a1ff-69f5189b5398","Type":"ContainerStarted","Data":"71e32910c03b5ab9dd51a7c1b7da260bc8b4431907e35f53f6433b7cedbbde97"}
Jul 16 21:48:30.317619 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.317597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6849m" event={"ID":"6c1ef0ce-398e-4a2a-99f0-071806905ba4","Type":"ContainerStarted","Data":"06f83a863b6c8a24597a535eb7a0bc090fdb9a1a2e206a637911c11b18a9f260"}
Jul 16 21:48:30.318901 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.318873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vdlb" event={"ID":"8b7bf600-8418-4011-85ea-de76f6ee0ea3","Type":"ContainerStarted","Data":"69ffa648d9027d43f7f2e3fce7aa22dcaec3145ad0af2e994ad4254ee277f8f2"}
Jul 16 21:48:30.320013 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.319987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb" event={"ID":"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2","Type":"ContainerStarted","Data":"1d5cd83c8fe2132884dd59e6dcbd23f78b3d7d8bbd9b4f92a494ce31242f857f"}
Jul 16 21:48:30.321341 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.321315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" event={"ID":"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d","Type":"ContainerStarted","Data":"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"}
Jul 16 21:48:30.321442 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.321359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:30.321442 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.321380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" event={"ID":"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d","Type":"ContainerStarted","Data":"0a42e0cc8fb0882843279285dc3eb476509c71840d57dcf54244c2a660470a92"}
Jul 16 21:48:30.322581 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.322555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhcbp" event={"ID":"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3","Type":"ContainerStarted","Data":"42e2e2ff816f4445ca54c108efe643077550c9da03d8e118b7c8dedbea9dc3b8"}
Jul 16 21:48:30.334327 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.334272    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x5mxz" podStartSLOduration=68.33424599 podStartE2EDuration="1m8.33424599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:30.333194619 +0000 UTC m=+68.843061692" watchObservedRunningTime="2026-07-16 21:48:30.33424599 +0000 UTC m=+68.844113073"
Jul 16 21:48:30.358036 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.357973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" podStartSLOduration=68.357904789 podStartE2EDuration="1m8.357904789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:30.356553911 +0000 UTC m=+68.866420993" watchObservedRunningTime="2026-07-16 21:48:30.357904789 +0000 UTC m=+68.867771852"
Jul 16 21:48:30.386146 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.386046    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k" podStartSLOduration=33.386031722 podStartE2EDuration="33.386031722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:30.384809899 +0000 UTC m=+68.894676980" watchObservedRunningTime="2026-07-16 21:48:30.386031722 +0000 UTC m=+68.895898782"
Jul 16 21:48:30.406795 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.406719    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-swksv" podStartSLOduration=24.564427822 podStartE2EDuration="33.406698603s" totalImagesPullingTime="8.842270781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:30.404642228 +0000 UTC m=+68.914509309" watchObservedRunningTime="2026-07-16 21:48:30.406698603 +0000 UTC m=+68.916565685"
Jul 16 21:48:30.419953 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.419922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-swksv"
Jul 16 21:48:30.570234 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.570182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:30.573255 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:30.573208    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:31.333884 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:31.332731    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:31.334497 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:31.334301    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6ddb94bc54-9kx9k"
Jul 16 21:48:39.363280 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.363167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhcbp" event={"ID":"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3","Type":"ContainerStarted","Data":"53926fbd7bbc7d1b763c4339153b9b32602bcdb0a9504e919fc4c9351d1bcc9e"}
Jul 16 21:48:39.363793 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.363282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhcbp" event={"ID":"9a4ec513-a38b-4c89-84f8-4992b1d4fcf3","Type":"ContainerStarted","Data":"5c48149a434b9e4031cbb27de0ed87159932dd3b000e933ed25c230bb047de18"}
Jul 16 21:48:39.364775 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.364749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d" event={"ID":"9b2c8f8c-0c6f-4891-b65b-b7d004450d38","Type":"ContainerStarted","Data":"7f3fbc41516c2451914fef6317ad2cdfd39022a71bb43fadfaf1d2cb230dd99c"}
Jul 16 21:48:39.366263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.366233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mj2sj" event={"ID":"9aa83795-0672-4fc2-9607-bbcc4434ece5","Type":"ContainerStarted","Data":"8b2b40ca9e4d49314597d41d299dd663f4b1283fe1afc4dce5245288bb79ac84"}
Jul 16 21:48:39.367901 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.367876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6849m" event={"ID":"6c1ef0ce-398e-4a2a-99f0-071806905ba4","Type":"ContainerStarted","Data":"d4b6ed5a9062a1cfa897303dc897530c69f0fd3c0bb6f9d03c2df1968b11e9f5"}
Jul 16 21:48:39.369563 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.369540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vdlb" event={"ID":"8b7bf600-8418-4011-85ea-de76f6ee0ea3","Type":"ContainerStarted","Data":"605a66ee91afe2c7499f31967b7ffe6befd6cafc82959fd645b4c86f48ecb991"}
Jul 16 21:48:39.369672 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.369586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vdlb" event={"ID":"8b7bf600-8418-4011-85ea-de76f6ee0ea3","Type":"ContainerStarted","Data":"98f08b93679895a853c41600fc1ed2f72f2293b8a1cdecf8b615effad7bdb338"}
Jul 16 21:48:39.369672 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.369635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:39.371493 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.371469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb" event={"ID":"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2","Type":"ContainerStarted","Data":"0e418c46221dcc420d70b6c114119049afed722f364a3e807f97b40cf16b47ff"}
Jul 16 21:48:39.371602 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.371509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb" event={"ID":"b0250bcf-5c0e-4db0-8f8d-fe6e1345a4d2","Type":"ContainerStarted","Data":"c87b5ee7f39b7ac6a382bedc1ff5cfe92795cdb6a3714c5aa4bc1b8ab9cf1d40"}
Jul 16 21:48:39.402376 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.402313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qhcbp" podStartSLOduration=69.655621985 podStartE2EDuration="1m17.402290175s" totalImagesPullingTime="7.74666819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.401086645 +0000 UTC m=+77.910953728" watchObservedRunningTime="2026-07-16 21:48:39.402290175 +0000 UTC m=+77.912157258"
Jul 16 21:48:39.466295 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.466231    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-ss5wb" podStartSLOduration=34.953107676 podStartE2EDuration="43.46621402s" totalImagesPullingTime="8.513106344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.434480204 +0000 UTC m=+77.944347281" watchObservedRunningTime="2026-07-16 21:48:39.46621402 +0000 UTC m=+77.976081112"
Jul 16 21:48:39.467119 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.467091    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6vdlb" podStartSLOduration=33.913531184 podStartE2EDuration="42.467080023s" totalImagesPullingTime="8.553548839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.463007142 +0000 UTC m=+77.972874223" watchObservedRunningTime="2026-07-16 21:48:39.467080023 +0000 UTC m=+77.976947106"
Jul 16 21:48:39.491784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.491730    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-lx69d" podStartSLOduration=33.746116375 podStartE2EDuration="42.491714247s" totalImagesPullingTime="8.745597872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.489383358 +0000 UTC m=+77.999250441" watchObservedRunningTime="2026-07-16 21:48:39.491714247 +0000 UTC m=+78.001581372"
Jul 16 21:48:39.518515 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.518425    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mj2sj" podStartSLOduration=68.883357671 podStartE2EDuration="1m17.518401407s" totalImagesPullingTime="8.635043736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:39.517671609 +0000 UTC m=+78.027538691" watchObservedRunningTime="2026-07-16 21:48:39.518401407 +0000 UTC m=+78.028268493"
Jul 16 21:48:39.594912 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.594880    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:48:39.597523 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.597505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.602851 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.602817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:48:39.602994 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.602817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:48:39.602994 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.602937    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:48:39.602994 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.602957    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fsgdm\""
Jul 16 21:48:39.603137 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.603041    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:48:39.603206 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.603190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:48:39.603353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.603334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:48:39.603407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.603369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:48:39.613221 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.613193    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:48:39.747361 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.747573 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.747573 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.747573 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrqm\" (UniqueName: \"kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.747573 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.747755 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.747581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.848857 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.848818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849026 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.848870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849026 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.848897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849157 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849157 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrqm\" (UniqueName: \"kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849253 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849655 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849777 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.849826 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.849788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.851402 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.851385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.851608 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.851590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.873492 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.873447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrqm\" (UniqueName: \"kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm\") pod \"console-5cf97b9d7f-5z2jl\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") " pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:39.906511 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:39.906448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:40.066558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:40.066529    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:48:40.069521 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:40.069443    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70165c69_cb61_48df_9087_8e17e14ee7fc.slice/crio-88eee4e19e28f9c00752d47c9aee7e6854fa425824fdc3a572eebd7db964f99c WatchSource:0}: Error finding container 88eee4e19e28f9c00752d47c9aee7e6854fa425824fdc3a572eebd7db964f99c: Status 404 returned error can't find the container with id 88eee4e19e28f9c00752d47c9aee7e6854fa425824fdc3a572eebd7db964f99c
Jul 16 21:48:40.378090 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:40.377983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf97b9d7f-5z2jl" event={"ID":"70165c69-cb61-48df-9087-8e17e14ee7fc","Type":"ContainerStarted","Data":"88eee4e19e28f9c00752d47c9aee7e6854fa425824fdc3a572eebd7db964f99c"}
Jul 16 21:48:43.469135 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.469100    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-574rp"]
Jul 16 21:48:43.471717 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.471688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.474902 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.474878    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 21:48:43.474998 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.474908    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 21:48:43.475907 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.475883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:48:43.476190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.476169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2xl6f\""
Jul 16 21:48:43.490877 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.490855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-574rp"]
Jul 16 21:48:43.582249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.582216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6768fca8-cba6-46e4-9b94-25f48aa790a1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.582249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.582266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9l9\" (UniqueName: \"kubernetes.io/projected/6768fca8-cba6-46e4-9b94-25f48aa790a1-kube-api-access-wp9l9\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.582517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.582299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.582517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.582380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.683479 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.683423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6768fca8-cba6-46e4-9b94-25f48aa790a1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.683665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.683512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9l9\" (UniqueName: \"kubernetes.io/projected/6768fca8-cba6-46e4-9b94-25f48aa790a1-kube-api-access-wp9l9\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.683665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.683543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.683665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.683611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.684316 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.684275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6768fca8-cba6-46e4-9b94-25f48aa790a1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.686812 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.686781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.686931 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.686878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6768fca8-cba6-46e4-9b94-25f48aa790a1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.702885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.702854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9l9\" (UniqueName: \"kubernetes.io/projected/6768fca8-cba6-46e4-9b94-25f48aa790a1-kube-api-access-wp9l9\") pod \"prometheus-operator-5945b5cfd-574rp\" (UID: \"6768fca8-cba6-46e4-9b94-25f48aa790a1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:43.783125 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:43.783084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp"
Jul 16 21:48:44.341410 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.341376    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-574rp"]
Jul 16 21:48:44.344079 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:44.344049    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6768fca8_cba6_46e4_9b94_25f48aa790a1.slice/crio-9aeff4289f5b15d97f46d485086ce5437045b29099015ffcfaca1e0894e72d34 WatchSource:0}: Error finding container 9aeff4289f5b15d97f46d485086ce5437045b29099015ffcfaca1e0894e72d34: Status 404 returned error can't find the container with id 9aeff4289f5b15d97f46d485086ce5437045b29099015ffcfaca1e0894e72d34
Jul 16 21:48:44.396906 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.396872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf97b9d7f-5z2jl" event={"ID":"70165c69-cb61-48df-9087-8e17e14ee7fc","Type":"ContainerStarted","Data":"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"}
Jul 16 21:48:44.398124 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.398097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp" event={"ID":"6768fca8-cba6-46e4-9b94-25f48aa790a1","Type":"ContainerStarted","Data":"9aeff4289f5b15d97f46d485086ce5437045b29099015ffcfaca1e0894e72d34"}
Jul 16 21:48:44.400059 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.400037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6849m" event={"ID":"6c1ef0ce-398e-4a2a-99f0-071806905ba4","Type":"ContainerStarted","Data":"14bcae91bee0211838beee895aa45a105a2a93490a5d36980524280bcec4389f"}
Jul 16 21:48:44.454335 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.454215    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cf97b9d7f-5z2jl" podStartSLOduration=1.310166479 podStartE2EDuration="5.454198331s" totalImagesPullingTime="4.144031852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:44.422435416 +0000 UTC m=+82.932302497" watchObservedRunningTime="2026-07-16 21:48:44.454198331 +0000 UTC m=+82.964065413"
Jul 16 21:48:44.454653 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:44.454624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6849m" podStartSLOduration=17.219189697 podStartE2EDuration="36.454613819s" totalImagesPullingTime="19.235424122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:44.452713718 +0000 UTC m=+82.962580802" watchObservedRunningTime="2026-07-16 21:48:44.454613819 +0000 UTC m=+82.964480900"
Jul 16 21:48:46.409567 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:46.409527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp" event={"ID":"6768fca8-cba6-46e4-9b94-25f48aa790a1","Type":"ContainerStarted","Data":"478e37712724ade6bdc990c0a3bc71cd3c32d0b890d034d2d5bb70bbee20afae"}
Jul 16 21:48:46.409962 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:46.409594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp" event={"ID":"6768fca8-cba6-46e4-9b94-25f48aa790a1","Type":"ContainerStarted","Data":"ecf55267e89d96e44ae7e85599447bc49b04ad1cc083649da0facdffa0a60ba4"}
Jul 16 21:48:46.446529 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:46.446467    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-574rp" podStartSLOduration=2.190968886 podStartE2EDuration="3.446435784s" totalImagesPullingTime="1.255466898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:46.444391383 +0000 UTC m=+84.954258465" watchObservedRunningTime="2026-07-16 21:48:46.446435784 +0000 UTC m=+84.956302866"
Jul 16 21:48:48.932352 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.932320    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"]
Jul 16 21:48:48.934708 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.934689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:48.938601 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.938578    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gfzrv\""
Jul 16 21:48:48.938796 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.938779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 16 21:48:48.938896 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.938811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:48:48.957908 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.957878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"]
Jul 16 21:48:48.981822 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.981788    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-swxjg"]
Jul 16 21:48:48.984738 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.984711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:48.994927 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.994491    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:48:48.994927 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.994491    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:48:48.994927 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.994820    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w8w4k\""
Jul 16 21:48:48.997554 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:48.997532    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:48:49.028749 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.028711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a602cf-12f5-42e2-a053-dda828f0b9e7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.028749 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.028753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.029001 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.028848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hcl\" (UniqueName: \"kubernetes.io/projected/a9a602cf-12f5-42e2-a053-dda828f0b9e7-kube-api-access-69hcl\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.029001 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.028928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.129595 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a602cf-12f5-42e2-a053-dda828f0b9e7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.129781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.129781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.129781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-wtmp\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.129781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-root\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.129781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5sl\" (UniqueName: \"kubernetes.io/projected/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-kube-api-access-kt5sl\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-sys\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hcl\" (UniqueName: \"kubernetes.io/projected/a9a602cf-12f5-42e2-a053-dda828f0b9e7-kube-api-access-69hcl\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-textfile\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.129953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-tls\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.130007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-metrics-client-ca\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.130043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.130039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.130395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.130256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a602cf-12f5-42e2-a053-dda828f0b9e7-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.132165 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.132137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.132292 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.132195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9a602cf-12f5-42e2-a053-dda828f0b9e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.139265 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.139242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hcl\" (UniqueName: \"kubernetes.io/projected/a9a602cf-12f5-42e2-a053-dda828f0b9e7-kube-api-access-69hcl\") pod \"openshift-state-metrics-79f6958cdf-cp4ws\" (UID: \"a9a602cf-12f5-42e2-a053-dda828f0b9e7\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.231322 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231322 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-textfile\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231322 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-tls\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-metrics-client-ca\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-wtmp\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-root\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5sl\" (UniqueName: \"kubernetes.io/projected/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-kube-api-access-kt5sl\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-sys\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231550    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-root\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231618 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-sys\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231893 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-wtmp\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.231893 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.231648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-textfile\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.232197 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.232174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.232268 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.232252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-metrics-client-ca\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.233682 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.233661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.233863 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.233846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-node-exporter-tls\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.242906 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.242884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5sl\" (UniqueName: \"kubernetes.io/projected/f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc-kube-api-access-kt5sl\") pod \"node-exporter-swxjg\" (UID: \"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc\") " pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.243707 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.243688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"
Jul 16 21:48:49.293339 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.293047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swxjg"
Jul 16 21:48:49.306144 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:49.306108    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96b41f0_3a0e_41fc_b7fe_ce62a32fa1cc.slice/crio-d20b2eca582338ec2a72f8777e4ca784149fd025f2808c14ac529f15bb73e971 WatchSource:0}: Error finding container d20b2eca582338ec2a72f8777e4ca784149fd025f2808c14ac529f15bb73e971: Status 404 returned error can't find the container with id d20b2eca582338ec2a72f8777e4ca784149fd025f2808c14ac529f15bb73e971
Jul 16 21:48:49.379805 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.379778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vdlb"
Jul 16 21:48:49.386632 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.386604    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws"]
Jul 16 21:48:49.390136 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:49.390111    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a602cf_12f5_42e2_a053_dda828f0b9e7.slice/crio-b205100706abd1b1929ff4b4d2a7b1b1a30f6ffa0e5e3f238cb69ed9f4e41ecd WatchSource:0}: Error finding container b205100706abd1b1929ff4b4d2a7b1b1a30f6ffa0e5e3f238cb69ed9f4e41ecd: Status 404 returned error can't find the container with id b205100706abd1b1929ff4b4d2a7b1b1a30f6ffa0e5e3f238cb69ed9f4e41ecd
Jul 16 21:48:49.423182 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.423145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws" event={"ID":"a9a602cf-12f5-42e2-a053-dda828f0b9e7","Type":"ContainerStarted","Data":"b205100706abd1b1929ff4b4d2a7b1b1a30f6ffa0e5e3f238cb69ed9f4e41ecd"}
Jul 16 21:48:49.424697 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.424666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swxjg" event={"ID":"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc","Type":"ContainerStarted","Data":"d20b2eca582338ec2a72f8777e4ca784149fd025f2808c14ac529f15bb73e971"}
Jul 16 21:48:49.906815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.906734    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:49.906815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.906805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:49.913277 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:49.913247    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:50.431600 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:50.431547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws" event={"ID":"a9a602cf-12f5-42e2-a053-dda828f0b9e7","Type":"ContainerStarted","Data":"2116bbc323671b51d461237c048be308d8b377d8332cb25e36a435bdfbc09337"}
Jul 16 21:48:50.432057 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:50.431627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws" event={"ID":"a9a602cf-12f5-42e2-a053-dda828f0b9e7","Type":"ContainerStarted","Data":"785d90b32a5420b1b397f5630d9fffdcaacab37853eb31cf83d03c9491d38e18"}
Jul 16 21:48:50.436115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:50.436088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:48:51.192506 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.192426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"]
Jul 16 21:48:51.195022 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.195003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.198407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.198384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-svx8c\""
Jul 16 21:48:51.214621 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.214600    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 21:48:51.215189 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.215171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 21:48:51.215449 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.215435    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c57puf4u1g46q\""
Jul 16 21:48:51.235527 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.235504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 21:48:51.236201 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.236186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 21:48:51.236395 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.236374    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 21:48:51.237665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.237647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"]
Jul 16 21:48:51.337622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.337597    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:48:51.350632 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350669    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-grpc-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-metrics-client-ca\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.350865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.350845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb822\" (UniqueName: \"kubernetes.io/projected/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-kube-api-access-rb822\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.437405 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.437365    2566 generic.go:356] "Generic (PLEG): container finished" podID="f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc" containerID="3eaf5746aa62e75df1b51147dc11c58221c89f7f5b4a8298575ff4abfb4a4a9c" exitCode=0
Jul 16 21:48:51.437879 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.437442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swxjg" event={"ID":"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc","Type":"ContainerDied","Data":"3eaf5746aa62e75df1b51147dc11c58221c89f7f5b4a8298575ff4abfb4a4a9c"}
Jul 16 21:48:51.439938 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.439916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws" event={"ID":"a9a602cf-12f5-42e2-a053-dda828f0b9e7","Type":"ContainerStarted","Data":"b54700569f398398444c323cb8aaaf9e688c6295958bcc6551d682ec245d4d1e"}
Jul 16 21:48:51.452001 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.451976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-grpc-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452111 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452111 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-metrics-client-ca\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452111 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb822\" (UniqueName: \"kubernetes.io/projected/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-kube-api-access-rb822\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452267 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452267 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452267 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.452412 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.452311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.453596 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.453548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-metrics-client-ca\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.455100 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.455074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.455100 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.455084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.455215 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.455130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.455876 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.455854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.455954 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.455911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-grpc-tls\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.456072 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.456049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.469539 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.469518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb822\" (UniqueName: \"kubernetes.io/projected/348f7975-0a2f-4c9f-98d5-e901e6cc75d8-kube-api-access-rb822\") pod \"thanos-querier-798fb6d8b5-xmcng\" (UID: \"348f7975-0a2f-4c9f-98d5-e901e6cc75d8\") " pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.499851 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.499806    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cp4ws" podStartSLOduration=2.403584735 podStartE2EDuration="3.499787934s" totalImagesPullingTime="1.096203199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:51.49823337 +0000 UTC m=+90.008100453" watchObservedRunningTime="2026-07-16 21:48:51.499787934 +0000 UTC m=+90.009655018"
Jul 16 21:48:51.503772 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.503751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:51.674047 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:51.673866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"]
Jul 16 21:48:51.676341 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:51.676315    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348f7975_0a2f_4c9f_98d5_e901e6cc75d8.slice/crio-3f8f11c5891244faeb02ba47849e1590a31dc79ba2fa04af7149c61b5897e126 WatchSource:0}: Error finding container 3f8f11c5891244faeb02ba47849e1590a31dc79ba2fa04af7149c61b5897e126: Status 404 returned error can't find the container with id 3f8f11c5891244faeb02ba47849e1590a31dc79ba2fa04af7149c61b5897e126
Jul 16 21:48:52.445781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:52.445738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swxjg" event={"ID":"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc","Type":"ContainerStarted","Data":"00e3b6b8e8c9e07bb6e5c30839c88e8d6200bfa1a7037d96f3aa1fd6586d7f6f"}
Jul 16 21:48:52.446263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:52.445806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swxjg" event={"ID":"f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc","Type":"ContainerStarted","Data":"348da446a127ec1bc2ec23b9557ad890cef7eb42baf7787292cdb2a1ee7475a8"}
Jul 16 21:48:52.447012 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:52.446984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"3f8f11c5891244faeb02ba47849e1590a31dc79ba2fa04af7149c61b5897e126"}
Jul 16 21:48:52.493162 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:52.493103    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-swxjg" podStartSLOduration=3.135553691 podStartE2EDuration="4.493083191s" totalImagesPullingTime="1.291134239s" totalInitContainerRuntime="66.395261ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:52.492159432 +0000 UTC m=+91.002026552" watchObservedRunningTime="2026-07-16 21:48:52.493083191 +0000 UTC m=+91.002950273"
Jul 16 21:48:53.708161 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.708120    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"]
Jul 16 21:48:53.711201 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.711178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.714828 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.714803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nmtps\""
Jul 16 21:48:53.715086 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.715067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 21:48:53.729187 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.729161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"]
Jul 16 21:48:53.774212 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.774173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9a8c775-a2f3-4838-a4fb-595f042e4493-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.774392 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.774222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6f6j\" (UniqueName: \"kubernetes.io/projected/c9a8c775-a2f3-4838-a4fb-595f042e4493-kube-api-access-q6f6j\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.875632 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.875594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9a8c775-a2f3-4838-a4fb-595f042e4493-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.875784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.875640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6f6j\" (UniqueName: \"kubernetes.io/projected/c9a8c775-a2f3-4838-a4fb-595f042e4493-kube-api-access-q6f6j\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.878197 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.878169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9a8c775-a2f3-4838-a4fb-595f042e4493-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:53.886080 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:53.886056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6f6j\" (UniqueName: \"kubernetes.io/projected/c9a8c775-a2f3-4838-a4fb-595f042e4493-kube-api-access-q6f6j\") pod \"monitoring-plugin-7b7b7bc8bf-n9ngj\" (UID: \"c9a8c775-a2f3-4838-a4fb-595f042e4493\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:54.021547 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.021520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:54.171705 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.171675    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"]
Jul 16 21:48:54.175066 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:48:54.175038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a8c775_a2f3_4838_a4fb_595f042e4493.slice/crio-46c3f6ed41608fe040236270aac0a2e2ca047dcc0ffd3b20314ece93e3225b89 WatchSource:0}: Error finding container 46c3f6ed41608fe040236270aac0a2e2ca047dcc0ffd3b20314ece93e3225b89: Status 404 returned error can't find the container with id 46c3f6ed41608fe040236270aac0a2e2ca047dcc0ffd3b20314ece93e3225b89
Jul 16 21:48:54.457857 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.457820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"f2102a0a027d58fc5a5127c50e42e67bc1e2060dc07b05f05ca0339bd2a993c6"}
Jul 16 21:48:54.458046 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.457895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"abefc35534dd421d7da52e57d0f211b8e3492d348684cb75ed6f96e5ad961b02"}
Jul 16 21:48:54.458046 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.457937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"451db26cda1d03713f6df1147262e34528adb1f8ca3e88a69adb45d3012c9ca5"}
Jul 16 21:48:54.458933 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:54.458909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj" event={"ID":"c9a8c775-a2f3-4838-a4fb-595f042e4493","Type":"ContainerStarted","Data":"46c3f6ed41608fe040236270aac0a2e2ca047dcc0ffd3b20314ece93e3225b89"}
Jul 16 21:48:55.467566 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:55.467525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"a51ef66f25144680246bad4d49a4501533fa98b371a397da41b896ab005bdaa1"}
Jul 16 21:48:55.467991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:55.467609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"7e39163025bd58a260074a4fc718055a68c0383622e540f54c112e81552a4895"}
Jul 16 21:48:55.467991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:55.467636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" event={"ID":"348f7975-0a2f-4c9f-98d5-e901e6cc75d8","Type":"ContainerStarted","Data":"e35d845a3c53722f391ca2bc3dd97ad454f272345ecc6d79e78881f16be819e9"}
Jul 16 21:48:55.467991 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:55.467762    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:48:55.515639 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:55.515587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng" podStartSLOduration=1.457407893 podStartE2EDuration="4.515571642s" totalImagesPullingTime="3.058163749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:55.514932086 +0000 UTC m=+94.024799169" watchObservedRunningTime="2026-07-16 21:48:55.515571642 +0000 UTC m=+94.025438765"
Jul 16 21:48:56.473143 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:56.473098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj" event={"ID":"c9a8c775-a2f3-4838-a4fb-595f042e4493","Type":"ContainerStarted","Data":"24779e35582eea8dc25bd1f7512c01efb62abda85e93c916038e606762b4e5a2"}
Jul 16 21:48:56.473613 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:56.473160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:56.477682 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:56.477660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj"
Jul 16 21:48:56.490910 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:56.490866    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-n9ngj" podStartSLOduration=1.732448449 podStartE2EDuration="3.490849903s" totalImagesPullingTime="1.758401454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:56.48994374 +0000 UTC m=+94.999810822" watchObservedRunningTime="2026-07-16 21:48:56.490849903 +0000 UTC m=+95.000716985"
Jul 16 21:48:57.269543 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:48:57.269511    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:49:01.334382 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:01.334351    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x5mxz"
Jul 16 21:49:01.479188 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:01.479162    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-798fb6d8b5-xmcng"
Jul 16 21:49:04.589536 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:04.589504    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:49:22.294473 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.294417    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" podUID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" containerName="registry" containerID="cri-o://bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962" gracePeriod=30
Jul 16 21:49:22.540361 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.540339    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:49:22.578261 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.578190    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" containerID="bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962" exitCode=0
Jul 16 21:49:22.578370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.578258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" event={"ID":"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d","Type":"ContainerDied","Data":"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"}
Jul 16 21:49:22.578370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.578304    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx"
Jul 16 21:49:22.578370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.578343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c74d96db6-gzjmx" event={"ID":"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d","Type":"ContainerDied","Data":"0a42e0cc8fb0882843279285dc3eb476509c71840d57dcf54244c2a660470a92"}
Jul 16 21:49:22.578569 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.578372    2566 scope.go:117] "RemoveContainer" containerID="bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"
Jul 16 21:49:22.588320 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.588292    2566 scope.go:117] "RemoveContainer" containerID="bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"
Jul 16 21:49:22.588595 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:49:22.588571    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962\": container with ID starting with bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962 not found: ID does not exist" containerID="bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"
Jul 16 21:49:22.588653 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.588608    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962"} err="failed to get container status \"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962\": rpc error: code = NotFound desc = could not find container \"bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962\": container with ID starting with bbb22b4154cddd3fac4becd31fe804d1648af9fd920be56677bb18827f435962 not found: ID does not exist"
Jul 16 21:49:22.624898 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.624870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.624982 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.624924    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.624982 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.624957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm27q\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.624999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625197 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625289 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625365 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets\") pod \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\" (UID: \"e3e9fbb1-79cb-4db4-972d-2c6f35778a1d\") "
Jul 16 21:49:22.625365 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:22.625545 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:22.625633 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.625582    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-trusted-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.627564 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.627522    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q" (OuterVolumeSpecName: "kube-api-access-cm27q") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "kube-api-access-cm27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:22.627658 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.627629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:22.627901 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.627867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:22.627997 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.627943    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:22.627997 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.627961    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:22.634098 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.634063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" (UID: "e3e9fbb1-79cb-4db4-972d-2c6f35778a1d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:49:22.726389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726345    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726388    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-installation-pull-secrets\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726407    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-image-registry-private-configuration\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726423    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm27q\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-kube-api-access-cm27q\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726435    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-registry-certificates\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726446    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-bound-sa-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.726587 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.726479    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d-ca-trust-extracted\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:22.902099 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.902070    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:49:22.910326 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:22.910300    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c74d96db6-gzjmx"]
Jul 16 21:49:24.010030 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:24.009992    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" path="/var/lib/kubelet/pods/e3e9fbb1-79cb-4db4-972d-2c6f35778a1d/volumes"
Jul 16 21:49:28.603428 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:28.603389    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1743492-997c-412c-8691-ec0bdfceec02" containerID="b7a1c581b7afc3e32161c9428d5e9b627be127fede64b82fc1e87f68063e9169" exitCode=0
Jul 16 21:49:28.603828 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:28.603479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z" event={"ID":"d1743492-997c-412c-8691-ec0bdfceec02","Type":"ContainerDied","Data":"b7a1c581b7afc3e32161c9428d5e9b627be127fede64b82fc1e87f68063e9169"}
Jul 16 21:49:28.603828 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:28.603795    2566 scope.go:117] "RemoveContainer" containerID="b7a1c581b7afc3e32161c9428d5e9b627be127fede64b82fc1e87f68063e9169"
Jul 16 21:49:29.609092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.609058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-d7q6z" event={"ID":"d1743492-997c-412c-8691-ec0bdfceec02","Type":"ContainerStarted","Data":"13240ac836320a70009ba1ac984d0b43106b7bc5e9334ccec28a87a22e881378"}
Jul 16 21:49:29.610264 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.610238    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cf97b9d7f-5z2jl" podUID="70165c69-cb61-48df-9087-8e17e14ee7fc" containerName="console" containerID="cri-o://8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529" gracePeriod=15
Jul 16 21:49:29.862092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.862039    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:49:29.987150 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrqm\" (UniqueName: \"kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config\") pod \"70165c69-cb61-48df-9087-8e17e14ee7fc\" (UID: \"70165c69-cb61-48df-9087-8e17e14ee7fc\") "
Jul 16 21:49:29.987795 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config" (OuterVolumeSpecName: "console-config") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:29.987795 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:29.987974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.987794    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:49:29.989485 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.989438    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:29.989739 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.989712    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:49:29.989836 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:29.989758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm" (OuterVolumeSpecName: "kube-api-access-mzrqm") pod "70165c69-cb61-48df-9087-8e17e14ee7fc" (UID: "70165c69-cb61-48df-9087-8e17e14ee7fc"). InnerVolumeSpecName "kube-api-access-mzrqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:49:30.088767 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088735    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-oauth-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.088767 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088762    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-console-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.088767 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088771    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.088972 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088781    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrqm\" (UniqueName: \"kubernetes.io/projected/70165c69-cb61-48df-9087-8e17e14ee7fc-kube-api-access-mzrqm\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.088972 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088790    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70165c69-cb61-48df-9087-8e17e14ee7fc-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.088972 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.088798    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70165c69-cb61-48df-9087-8e17e14ee7fc-console-oauth-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:49:30.614699 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.614663    2566 generic.go:356] "Generic (PLEG): container finished" podID="70165c69-cb61-48df-9087-8e17e14ee7fc" containerID="8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529" exitCode=0
Jul 16 21:49:30.615115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.614730    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf97b9d7f-5z2jl"
Jul 16 21:49:30.615115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.614733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf97b9d7f-5z2jl" event={"ID":"70165c69-cb61-48df-9087-8e17e14ee7fc","Type":"ContainerDied","Data":"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"}
Jul 16 21:49:30.615115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.614865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf97b9d7f-5z2jl" event={"ID":"70165c69-cb61-48df-9087-8e17e14ee7fc","Type":"ContainerDied","Data":"88eee4e19e28f9c00752d47c9aee7e6854fa425824fdc3a572eebd7db964f99c"}
Jul 16 21:49:30.615115 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.614886    2566 scope.go:117] "RemoveContainer" containerID="8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"
Jul 16 21:49:30.622922 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.622901    2566 scope.go:117] "RemoveContainer" containerID="8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"
Jul 16 21:49:30.623169 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:49:30.623151    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529\": container with ID starting with 8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529 not found: ID does not exist" containerID="8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"
Jul 16 21:49:30.623218 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.623177    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529"} err="failed to get container status \"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529\": rpc error: code = NotFound desc = could not find container \"8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529\": container with ID starting with 8b826f3a918e11190f52c583133deb66700700b8935cfab1095533ecb8963529 not found: ID does not exist"
Jul 16 21:49:30.651945 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.651922    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:49:30.653109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:30.653090    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cf97b9d7f-5z2jl"]
Jul 16 21:49:32.010519 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:32.010488    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70165c69-cb61-48df-9087-8e17e14ee7fc" path="/var/lib/kubelet/pods/70165c69-cb61-48df-9087-8e17e14ee7fc/volumes"
Jul 16 21:49:33.629122 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:33.629088    2566 generic.go:356] "Generic (PLEG): container finished" podID="d3978b80-d899-4fa8-86b0-48885a155b24" containerID="93f1173ad3e57230970349ba5dea04041a9505f3e992454419a3df256f35b9a4" exitCode=0
Jul 16 21:49:33.629538 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:33.629154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8bbmz" event={"ID":"d3978b80-d899-4fa8-86b0-48885a155b24","Type":"ContainerDied","Data":"93f1173ad3e57230970349ba5dea04041a9505f3e992454419a3df256f35b9a4"}
Jul 16 21:49:33.629538 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:33.629481    2566 scope.go:117] "RemoveContainer" containerID="93f1173ad3e57230970349ba5dea04041a9505f3e992454419a3df256f35b9a4"
Jul 16 21:49:34.636759 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:34.636720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8bbmz" event={"ID":"d3978b80-d899-4fa8-86b0-48885a155b24","Type":"ContainerStarted","Data":"6efd21ec129200a0fe83c84b1df9001faa36881cae952d74b684ac7eb0347d25"}
Jul 16 21:49:38.653731 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:38.653697    2566 generic.go:356] "Generic (PLEG): container finished" podID="603d23ae-3670-4321-b0c3-15e9d77519e2" containerID="05cd9467702769638fca73fa20c6c5eab7bb94b62bb8b206c851bd15d7d372b7" exitCode=0
Jul 16 21:49:38.654137 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:38.653754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl" event={"ID":"603d23ae-3670-4321-b0c3-15e9d77519e2","Type":"ContainerDied","Data":"05cd9467702769638fca73fa20c6c5eab7bb94b62bb8b206c851bd15d7d372b7"}
Jul 16 21:49:38.654137 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:38.654060    2566 scope.go:117] "RemoveContainer" containerID="05cd9467702769638fca73fa20c6c5eab7bb94b62bb8b206c851bd15d7d372b7"
Jul 16 21:49:39.659815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:49:39.659779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-vjrnl" event={"ID":"603d23ae-3670-4321-b0c3-15e9d77519e2","Type":"ContainerStarted","Data":"308dfae48a341e59a5ec440cf52312b0c85a38cdac47c3b4dd2dc0af4a8fd4a2"}
Jul 16 21:50:18.521312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521219    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521616    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" containerName="registry"
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521632    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" containerName="registry"
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521642    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70165c69-cb61-48df-9087-8e17e14ee7fc" containerName="console"
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521648    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70165c69-cb61-48df-9087-8e17e14ee7fc" containerName="console"
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521711    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e9fbb1-79cb-4db4-972d-2c6f35778a1d" containerName="registry"
Jul 16 21:50:18.521781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.521719    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70165c69-cb61-48df-9087-8e17e14ee7fc" containerName="console"
Jul 16 21:50:18.524493 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.524472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.527707 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.527680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:50:18.527857 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.527831    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:50:18.527988 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.527709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:50:18.527988 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.527709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:50:18.528738 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.528717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:50:18.528940 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.528921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:50:18.529362 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.529341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:50:18.529448 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.529348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fsgdm\""
Jul 16 21:50:18.533864 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.533740    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:50:18.537109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.537087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:50:18.643535 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643535 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488nl\" (UniqueName: \"kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643730 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.643852 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.643754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744830 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488nl\" (UniqueName: \"kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.744984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.745216 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.744999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.745754 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.745726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.745868 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.745797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.745868 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.745806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.745868 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.745854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.747444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.747422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.747532 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.747498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.754894 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.754875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-488nl\" (UniqueName: \"kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl\") pod \"console-5b6cc9cc59-bz587\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") " pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.837017 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.836960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:18.970363 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:18.970329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:50:18.975823 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:50:18.975801    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccbbcef_bb84_4750_a206_3075d20ea3de.slice/crio-b4562179f64f38633ef4818f85110e7b76d412dc0a01c49be327c961570ef737 WatchSource:0}: Error finding container b4562179f64f38633ef4818f85110e7b76d412dc0a01c49be327c961570ef737: Status 404 returned error can't find the container with id b4562179f64f38633ef4818f85110e7b76d412dc0a01c49be327c961570ef737
Jul 16 21:50:19.822163 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:19.822125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6cc9cc59-bz587" event={"ID":"2ccbbcef-bb84-4750-a206-3075d20ea3de","Type":"ContainerStarted","Data":"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"}
Jul 16 21:50:19.822558 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:19.822210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6cc9cc59-bz587" event={"ID":"2ccbbcef-bb84-4750-a206-3075d20ea3de","Type":"ContainerStarted","Data":"b4562179f64f38633ef4818f85110e7b76d412dc0a01c49be327c961570ef737"}
Jul 16 21:50:19.845649 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:19.845589    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b6cc9cc59-bz587" podStartSLOduration=1.845574272 podStartE2EDuration="1.845574272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:19.844065694 +0000 UTC m=+178.353932776" watchObservedRunningTime="2026-07-16 21:50:19.845574272 +0000 UTC m=+178.355441355"
Jul 16 21:50:28.837659 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:28.837621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:28.838055 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:28.837699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:28.843120 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:28.843096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:50:28.858998 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:50:28.858977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:51:30.835295 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.835264    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:51:30.838658 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.838639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.862063 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.862038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:51:30.885351 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885488 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gvz\" (UniqueName: \"kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885633 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.885633 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.885565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.986851 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gvz\" (UniqueName: \"kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.986851 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987087 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987087 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987087 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987087 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.986961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987279 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.987087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987716 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.987693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987832 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.987733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.987832 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.987779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.988054 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.988033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.989436 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.989409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:30.989633 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:30.989613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:31.008112 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:31.008088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gvz\" (UniqueName: \"kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz\") pod \"console-5f8f99b575-dnkb6\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") " pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:31.147874 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:31.147795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:31.283782 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:31.283756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:51:31.285541 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:51:31.285515    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f55b345_edce_47dd_933f_ab2b7b5e7a7b.slice/crio-527b320cff95bb39141bdd2caeccd507ee3ee0c4e39dc387b32c9ff1ac37adc0 WatchSource:0}: Error finding container 527b320cff95bb39141bdd2caeccd507ee3ee0c4e39dc387b32c9ff1ac37adc0: Status 404 returned error can't find the container with id 527b320cff95bb39141bdd2caeccd507ee3ee0c4e39dc387b32c9ff1ac37adc0
Jul 16 21:51:32.106690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:32.106655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8f99b575-dnkb6" event={"ID":"8f55b345-edce-47dd-933f-ab2b7b5e7a7b","Type":"ContainerStarted","Data":"dda971a3e32b53dc27afb4d42a3192527daee5e34f542fc0ec9451eb513fd6da"}
Jul 16 21:51:32.107104 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:32.106704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8f99b575-dnkb6" event={"ID":"8f55b345-edce-47dd-933f-ab2b7b5e7a7b","Type":"ContainerStarted","Data":"527b320cff95bb39141bdd2caeccd507ee3ee0c4e39dc387b32c9ff1ac37adc0"}
Jul 16 21:51:41.148475 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:41.148428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:41.148924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:41.148503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:41.153066 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:41.153042    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:41.178674 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:41.178628    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f8f99b575-dnkb6" podStartSLOduration=11.178614984 podStartE2EDuration="11.178614984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:32.13557265 +0000 UTC m=+250.645439739" watchObservedRunningTime="2026-07-16 21:51:41.178614984 +0000 UTC m=+259.688482066"
Jul 16 21:51:42.155035 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:42.155003    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:51:42.217241 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:51:42.217206    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:52:07.238550 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.238485    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b6cc9cc59-bz587" podUID="2ccbbcef-bb84-4750-a206-3075d20ea3de" containerName="console" containerID="cri-o://17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee" gracePeriod=15
Jul 16 21:52:07.481988 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.481965    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:52:07.588348 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588348 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588546 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-488nl\" (UniqueName: \"kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588546 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588546 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588546 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588546 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588510    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert\") pod \"2ccbbcef-bb84-4750-a206-3075d20ea3de\" (UID: \"2ccbbcef-bb84-4750-a206-3075d20ea3de\") "
Jul 16 21:52:07.588881 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588836    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:07.588881 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config" (OuterVolumeSpecName: "console-config") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:07.589098 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588898    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:07.589098 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.588978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:07.590626 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.590604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:07.590696 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.590625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl" (OuterVolumeSpecName: "kube-api-access-488nl") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "kube-api-access-488nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:07.590696 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.590680    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ccbbcef-bb84-4750-a206-3075d20ea3de" (UID: "2ccbbcef-bb84-4750-a206-3075d20ea3de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:07.689242 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689208    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488nl\" (UniqueName: \"kubernetes.io/projected/2ccbbcef-bb84-4750-a206-3075d20ea3de-kube-api-access-488nl\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689242 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689235    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-oauth-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689242 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689246    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689439 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689254    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-oauth-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689439 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689263    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ccbbcef-bb84-4750-a206-3075d20ea3de-console-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689439 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689272    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:07.689439 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:07.689282    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ccbbcef-bb84-4750-a206-3075d20ea3de-trusted-ca-bundle\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:52:08.249667 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.249629    2566 generic.go:356] "Generic (PLEG): container finished" podID="2ccbbcef-bb84-4750-a206-3075d20ea3de" containerID="17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee" exitCode=0
Jul 16 21:52:08.250108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.249675    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6cc9cc59-bz587"
Jul 16 21:52:08.250108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.249681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6cc9cc59-bz587" event={"ID":"2ccbbcef-bb84-4750-a206-3075d20ea3de","Type":"ContainerDied","Data":"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"}
Jul 16 21:52:08.250108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.249732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6cc9cc59-bz587" event={"ID":"2ccbbcef-bb84-4750-a206-3075d20ea3de","Type":"ContainerDied","Data":"b4562179f64f38633ef4818f85110e7b76d412dc0a01c49be327c961570ef737"}
Jul 16 21:52:08.250108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.249759    2566 scope.go:117] "RemoveContainer" containerID="17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"
Jul 16 21:52:08.257760 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.257744    2566 scope.go:117] "RemoveContainer" containerID="17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"
Jul 16 21:52:08.257996 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:52:08.257980    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee\": container with ID starting with 17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee not found: ID does not exist" containerID="17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"
Jul 16 21:52:08.258054 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.258001    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee"} err="failed to get container status \"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee\": rpc error: code = NotFound desc = could not find container \"17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee\": container with ID starting with 17d04ac0f57da305c4941b746ac9f145cf32acbfe8b10dbde293f01d753e1bee not found: ID does not exist"
Jul 16 21:52:08.265545 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.265522    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:52:08.271776 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:08.271730    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b6cc9cc59-bz587"]
Jul 16 21:52:10.013529 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:10.013482    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ccbbcef-bb84-4750-a206-3075d20ea3de" path="/var/lib/kubelet/pods/2ccbbcef-bb84-4750-a206-3075d20ea3de/volumes"
Jul 16 21:52:21.887237 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:21.887206    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:52:21.890595 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:21.890571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:52:21.897409 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:21.897387    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:52:57.686632 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.686602    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pcbh7"]
Jul 16 21:52:57.688981 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.686911    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccbbcef-bb84-4750-a206-3075d20ea3de" containerName="console"
Jul 16 21:52:57.688981 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.686921    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccbbcef-bb84-4750-a206-3075d20ea3de" containerName="console"
Jul 16 21:52:57.688981 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.686974    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ccbbcef-bb84-4750-a206-3075d20ea3de" containerName="console"
Jul 16 21:52:57.689896 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.689868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.694555 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.694527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 21:52:57.695500 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.695450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qd5d7\""
Jul 16 21:52:57.695622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.695522    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 21:52:57.706996 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.706972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pcbh7"]
Jul 16 21:52:57.769783 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.769755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.769942 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.769799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2dv2\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-kube-api-access-g2dv2\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.870877 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.870847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.871016 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.870894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2dv2\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-kube-api-access-g2dv2\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.889691 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.889665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2dv2\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-kube-api-access-g2dv2\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:57.892239 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:57.892216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd54419e-f258-49bd-ae59-b253516443e5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pcbh7\" (UID: \"dd54419e-f258-49bd-ae59-b253516443e5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:58.006700 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:58.006672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:52:58.130671 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:58.130648    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pcbh7"]
Jul 16 21:52:58.133121 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:52:58.133091    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd54419e_f258_49bd_ae59_b253516443e5.slice/crio-944d4616f10e81dc4801926d3947861c2f0e66e832b657a75e55d96cc9c78231 WatchSource:0}: Error finding container 944d4616f10e81dc4801926d3947861c2f0e66e832b657a75e55d96cc9c78231: Status 404 returned error can't find the container with id 944d4616f10e81dc4801926d3947861c2f0e66e832b657a75e55d96cc9c78231
Jul 16 21:52:58.135108 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:58.135092    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:52:58.454580 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:52:58.454503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7" event={"ID":"dd54419e-f258-49bd-ae59-b253516443e5","Type":"ContainerStarted","Data":"944d4616f10e81dc4801926d3947861c2f0e66e832b657a75e55d96cc9c78231"}
Jul 16 21:53:02.475055 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:02.475020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7" event={"ID":"dd54419e-f258-49bd-ae59-b253516443e5","Type":"ContainerStarted","Data":"96387d73a7d5453dd25087c8bebb606bd1b97c8df3efceb687a8d95207e9e79a"}
Jul 16 21:53:02.475497 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:02.475084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:53:08.481089 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:08.481052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7"
Jul 16 21:53:08.498995 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:08.498945    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pcbh7" podStartSLOduration=8.150537665 podStartE2EDuration="11.498930616s" totalImagesPullingTime="3.348392951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:02.493513942 +0000 UTC m=+341.003381025" watchObservedRunningTime="2026-07-16 21:53:08.498930616 +0000 UTC m=+347.008797698"
Jul 16 21:53:11.082438 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.082401    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"]
Jul 16 21:53:11.085756 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.085738    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.088294 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.088265    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 21:53:11.089176 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.089159    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9hhs4\""
Jul 16 21:53:11.089247 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.089165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:53:11.094095 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.094075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"]
Jul 16 21:53:11.183383 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.183339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.183585 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.183421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf49k\" (UniqueName: \"kubernetes.io/projected/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-kube-api-access-vf49k\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.283890 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.283854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf49k\" (UniqueName: \"kubernetes.io/projected/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-kube-api-access-vf49k\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.284060 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.283901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.284222 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.284206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.292760 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.292739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf49k\" (UniqueName: \"kubernetes.io/projected/84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149-kube-api-access-vf49k\") pod \"openshift-lws-operator-fd8ccff4c-dm7dr\" (UID: \"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.394984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.394899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"
Jul 16 21:53:11.525885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:11.525803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr"]
Jul 16 21:53:11.528122 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:53:11.528098    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84c5d4b0_bbb7_4326_9d00_1f3a7bfb3149.slice/crio-224dd586c160deed14aa44973873c578d3595b08b1ad5f6a3ab1d5e798ae36c7 WatchSource:0}: Error finding container 224dd586c160deed14aa44973873c578d3595b08b1ad5f6a3ab1d5e798ae36c7: Status 404 returned error can't find the container with id 224dd586c160deed14aa44973873c578d3595b08b1ad5f6a3ab1d5e798ae36c7
Jul 16 21:53:12.515815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:12.515763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr" event={"ID":"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149","Type":"ContainerStarted","Data":"224dd586c160deed14aa44973873c578d3595b08b1ad5f6a3ab1d5e798ae36c7"}
Jul 16 21:53:14.525863 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:14.525823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr" event={"ID":"84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149","Type":"ContainerStarted","Data":"0248a0d1eddbd423ca08c4a963d523085145351e303f04ad4d10f2e40341f139"}
Jul 16 21:53:14.544684 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:14.544634    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dm7dr" podStartSLOduration=1.3009730369999999 podStartE2EDuration="3.544618938s" totalImagesPullingTime="2.243645901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:14.542776002 +0000 UTC m=+353.052643084" watchObservedRunningTime="2026-07-16 21:53:14.544618938 +0000 UTC m=+353.054486020"
Jul 16 21:53:34.489217 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.489184    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"]
Jul 16 21:53:34.501927 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.501902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.525667 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.525607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 21:53:34.525798 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.525728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 21:53:34.526071 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.526055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9vl5d\""
Jul 16 21:53:34.526308 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.526294    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 21:53:34.526470 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.526434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 21:53:34.527339 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.527321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"]
Jul 16 21:53:34.671902 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.671864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.671902 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.671902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xm84\" (UniqueName: \"kubernetes.io/projected/40c0974e-151e-40e5-a6c3-bad33f4c32b0-kube-api-access-8xm84\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.672163 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.671937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.672163 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.672005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.672163 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.672080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.773429 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.773429 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.773429 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xm84\" (UniqueName: \"kubernetes.io/projected/40c0974e-151e-40e5-a6c3-bad33f4c32b0-kube-api-access-8xm84\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.773736 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.773736 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.774003 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.773982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-manifests\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.775838 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.775816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40c0974e-151e-40e5-a6c3-bad33f4c32b0-tmp\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.776005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.775988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.776128 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.776106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40c0974e-151e-40e5-a6c3-bad33f4c32b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.785832 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.785810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xm84\" (UniqueName: \"kubernetes.io/projected/40c0974e-151e-40e5-a6c3-bad33f4c32b0-kube-api-access-8xm84\") pod \"opendatahub-operator-controller-manager-574666dc8f-5jrpx\" (UID: \"40c0974e-151e-40e5-a6c3-bad33f4c32b0\") " pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.811971 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.811943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:34.986705 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:34.986678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"]
Jul 16 21:53:34.988992 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:53:34.988964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c0974e_151e_40e5_a6c3_bad33f4c32b0.slice/crio-f00b573eaf8e852949708350129329d3e7f6f33e41de7a823df5611e022a3fda WatchSource:0}: Error finding container f00b573eaf8e852949708350129329d3e7f6f33e41de7a823df5611e022a3fda: Status 404 returned error can't find the container with id f00b573eaf8e852949708350129329d3e7f6f33e41de7a823df5611e022a3fda
Jul 16 21:53:35.610392 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:35.610350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx" event={"ID":"40c0974e-151e-40e5-a6c3-bad33f4c32b0","Type":"ContainerStarted","Data":"f00b573eaf8e852949708350129329d3e7f6f33e41de7a823df5611e022a3fda"}
Jul 16 21:53:38.628660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:38.628568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx" event={"ID":"40c0974e-151e-40e5-a6c3-bad33f4c32b0","Type":"ContainerStarted","Data":"295fed4759e42f62d08370438ea9b8dbd2b6eee700ce9bbfb5e9ce6d7dc0c23b"}
Jul 16 21:53:39.634676 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:39.634639    2566 generic.go:356] "Generic (PLEG): container finished" podID="40c0974e-151e-40e5-a6c3-bad33f4c32b0" containerID="295fed4759e42f62d08370438ea9b8dbd2b6eee700ce9bbfb5e9ce6d7dc0c23b" exitCode=0
Jul 16 21:53:39.635036 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:39.634710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx" event={"ID":"40c0974e-151e-40e5-a6c3-bad33f4c32b0","Type":"ContainerDied","Data":"295fed4759e42f62d08370438ea9b8dbd2b6eee700ce9bbfb5e9ce6d7dc0c23b"}
Jul 16 21:53:40.640679 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:40.640644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx" event={"ID":"40c0974e-151e-40e5-a6c3-bad33f4c32b0","Type":"ContainerStarted","Data":"c64686899567d045c7859dd4e896cdf08c9c633f4d7aff07b559ab960e5f58af"}
Jul 16 21:53:40.641051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:40.640805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:40.700270 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:40.700215    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx" podStartSLOduration=2.014175898 podStartE2EDuration="6.70019786s" totalImagesPullingTime="3.631958288s" totalInitContainerRuntime="1.054063674s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:40.699009453 +0000 UTC m=+379.208876535" watchObservedRunningTime="2026-07-16 21:53:40.70019786 +0000 UTC m=+379.210064941"
Jul 16 21:53:51.648263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:51.648233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-574666dc8f-5jrpx"
Jul 16 21:53:54.513533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.513499    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"]
Jul 16 21:53:54.516764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.516745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.528885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.528865    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 21:53:54.528974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.528872    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bdbwp\""
Jul 16 21:53:54.530002 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.529983    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 21:53:54.539375 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.539354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdc9\" (UniqueName: \"kubernetes.io/projected/d303fb66-62db-49a0-afce-89f3313d9ec6-kube-api-access-4tdc9\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.539474 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.539391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d303fb66-62db-49a0-afce-89f3313d9ec6-tls-certs\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.539474 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.539416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d303fb66-62db-49a0-afce-89f3313d9ec6-tmp\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.553924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.553900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"]
Jul 16 21:53:54.640756 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.640729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdc9\" (UniqueName: \"kubernetes.io/projected/d303fb66-62db-49a0-afce-89f3313d9ec6-kube-api-access-4tdc9\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.640910 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.640770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d303fb66-62db-49a0-afce-89f3313d9ec6-tls-certs\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.640910 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.640799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d303fb66-62db-49a0-afce-89f3313d9ec6-tmp\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.643062 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.643030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d303fb66-62db-49a0-afce-89f3313d9ec6-tmp\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.643174 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.643164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d303fb66-62db-49a0-afce-89f3313d9ec6-tls-certs\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.670841 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.670806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdc9\" (UniqueName: \"kubernetes.io/projected/d303fb66-62db-49a0-afce-89f3313d9ec6-kube-api-access-4tdc9\") pod \"kube-auth-proxy-54f57477db-2tn7p\" (UID: \"d303fb66-62db-49a0-afce-89f3313d9ec6\") " pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.706308 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.706272    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nlx76"]
Jul 16 21:53:54.710233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.710204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.715678 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.715652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 16 21:53:54.715789 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.715745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hs4h8\""
Jul 16 21:53:54.736183 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.736156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nlx76"]
Jul 16 21:53:54.741510 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.741484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r58\" (UniqueName: \"kubernetes.io/projected/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-kube-api-access-s2r58\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.741631 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.741544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.825976 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.825902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"
Jul 16 21:53:54.842109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.842081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r58\" (UniqueName: \"kubernetes.io/projected/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-kube-api-access-s2r58\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.842244 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.842145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.842310 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:53:54.842289    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 16 21:53:54.842367 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:53:54.842355    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert podName:75ce86e8-4a0f-4bed-99cc-8a206c2216cf nodeName:}" failed. No retries permitted until 2026-07-16 21:53:55.342335947 +0000 UTC m=+393.852203011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert") pod "odh-model-controller-7998c67dc4-nlx76" (UID: "75ce86e8-4a0f-4bed-99cc-8a206c2216cf") : secret "odh-model-controller-webhook-cert" not found
Jul 16 21:53:54.878267 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.878233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r58\" (UniqueName: \"kubernetes.io/projected/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-kube-api-access-s2r58\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:54.984559 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:54.984525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f57477db-2tn7p"]
Jul 16 21:53:54.991901 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:53:54.991872    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd303fb66_62db_49a0_afce_89f3313d9ec6.slice/crio-123994d852b6d664430087ad200ad336cb9bf740ba2ba1d429cddad9b0c1c70b WatchSource:0}: Error finding container 123994d852b6d664430087ad200ad336cb9bf740ba2ba1d429cddad9b0c1c70b: Status 404 returned error can't find the container with id 123994d852b6d664430087ad200ad336cb9bf740ba2ba1d429cddad9b0c1c70b
Jul 16 21:53:55.346015 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:55.345980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:55.348343 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:55.348323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ce86e8-4a0f-4bed-99cc-8a206c2216cf-cert\") pod \"odh-model-controller-7998c67dc4-nlx76\" (UID: \"75ce86e8-4a0f-4bed-99cc-8a206c2216cf\") " pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:55.620501 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:55.620417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:55.717809 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:55.717770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p" event={"ID":"d303fb66-62db-49a0-afce-89f3313d9ec6","Type":"ContainerStarted","Data":"123994d852b6d664430087ad200ad336cb9bf740ba2ba1d429cddad9b0c1c70b"}
Jul 16 21:53:55.784843 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:53:55.784811    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ce86e8_4a0f_4bed_99cc_8a206c2216cf.slice/crio-6737ee27f1cf2c3e0a510b4222c3f2b96098dfc76691080cbf0f36bdb9a3d6c2 WatchSource:0}: Error finding container 6737ee27f1cf2c3e0a510b4222c3f2b96098dfc76691080cbf0f36bdb9a3d6c2: Status 404 returned error can't find the container with id 6737ee27f1cf2c3e0a510b4222c3f2b96098dfc76691080cbf0f36bdb9a3d6c2
Jul 16 21:53:55.791268 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:55.791244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nlx76"]
Jul 16 21:53:56.725835 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:56.725791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76" event={"ID":"75ce86e8-4a0f-4bed-99cc-8a206c2216cf","Type":"ContainerStarted","Data":"6737ee27f1cf2c3e0a510b4222c3f2b96098dfc76691080cbf0f36bdb9a3d6c2"}
Jul 16 21:53:57.045496 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.045443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:53:57.049665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.049632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.052525 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.052498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 21:53:57.053150 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.053128    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 21:53:57.053667 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.053499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gn6s5\""
Jul 16 21:53:57.053783 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.053710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 21:53:57.054033 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.053884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 21:53:57.059750 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059856 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059856 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtzd\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.059970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.059949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.071036 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.070963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:53:57.161064 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtzd\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.161771 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.161742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.163972 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.163952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.164204 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.164182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.164694 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.164672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.165207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.165183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.171124 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.171096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.171757 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.171738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtzd\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd\") pod \"istiod-openshift-gateway-7574459656-57tdn\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:57.364353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:57.364262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:53:58.967061 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:58.967033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:53:58.969054 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:53:58.969025    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f88e09_97c1_4f56_931e_9c7d7f770bbe.slice/crio-3baf28ef3e3f42e9669ac6f62810336461f77835fee670c065b839f967fd030c WatchSource:0}: Error finding container 3baf28ef3e3f42e9669ac6f62810336461f77835fee670c065b839f967fd030c: Status 404 returned error can't find the container with id 3baf28ef3e3f42e9669ac6f62810336461f77835fee670c065b839f967fd030c
Jul 16 21:53:59.744531 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.744495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" event={"ID":"86f88e09-97c1-4f56-931e-9c7d7f770bbe","Type":"ContainerStarted","Data":"3baf28ef3e3f42e9669ac6f62810336461f77835fee670c065b839f967fd030c"}
Jul 16 21:53:59.746242 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.746203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76" event={"ID":"75ce86e8-4a0f-4bed-99cc-8a206c2216cf","Type":"ContainerStarted","Data":"8454781c5c3b6d4f421d9963876e290efa35f050fd3038bee5b23b282cc9f686"}
Jul 16 21:53:59.746397 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.746325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:53:59.748001 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.747977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p" event={"ID":"d303fb66-62db-49a0-afce-89f3313d9ec6","Type":"ContainerStarted","Data":"17814902afab47df78edcfd29b1b266dd08379adb074c147f189bd61bf688c2b"}
Jul 16 21:53:59.775727 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.775659    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76" podStartSLOduration=2.684103112 podStartE2EDuration="5.775641414s" totalImagesPullingTime="3.091538302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:59.775371378 +0000 UTC m=+398.285238470" watchObservedRunningTime="2026-07-16 21:53:59.775641414 +0000 UTC m=+398.285508499"
Jul 16 21:53:59.814737 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:53:59.814667    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f57477db-2tn7p" podStartSLOduration=1.979891552 podStartE2EDuration="5.814646741s" totalImagesPullingTime="3.834755189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:59.810935939 +0000 UTC m=+398.320803020" watchObservedRunningTime="2026-07-16 21:53:59.814646741 +0000 UTC m=+398.324513828"
Jul 16 21:54:01.449888 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:01.449843    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:54:01.450266 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:01.449926    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:54:01.759245 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:01.759205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" event={"ID":"86f88e09-97c1-4f56-931e-9c7d7f770bbe","Type":"ContainerStarted","Data":"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"}
Jul 16 21:54:01.759422 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:01.759333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:54:02.766060 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:02.766023    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-57tdn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 21:54:02.766423 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:02.766077    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:54:04.402554 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.402507    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" podStartSLOduration=4.924202791 podStartE2EDuration="7.402489378s" totalImagesPullingTime="2.478286587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:01.797145347 +0000 UTC m=+400.307012450" watchObservedRunningTime="2026-07-16 21:54:04.402489378 +0000 UTC m=+402.912356460"
Jul 16 21:54:04.403489 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.403468    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"]
Jul 16 21:54:04.407052 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.407034    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.412616 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.412593    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tcgp5\""
Jul 16 21:54:04.434316 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.434288    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"]
Jul 16 21:54:04.531151 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531151 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34873d-40f2-4153-a382-d69f6c703405-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gxm\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-kube-api-access-j2gxm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531353 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.531565 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.531399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34873d-40f2-4153-a382-d69f6c703405-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632022 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.631986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gxm\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-kube-api-access-j2gxm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632190 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34873d-40f2-4153-a382-d69f6c703405-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632407 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34873d-40f2-4153-a382-d69f6c703405-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632646 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632646 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632646 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.632814 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.632787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.633093 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.633074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc34873d-40f2-4153-a382-d69f6c703405-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.634322 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.634296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc34873d-40f2-4153-a382-d69f6c703405-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.634740 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.634722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc34873d-40f2-4153-a382-d69f6c703405-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.653802 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.653750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.658788 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.658765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gxm\" (UniqueName: \"kubernetes.io/projected/dc34873d-40f2-4153-a382-d69f6c703405-kube-api-access-j2gxm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf47ptlz\" (UID: \"dc34873d-40f2-4153-a382-d69f6c703405\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.716592 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.716559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:04.920595 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:04.920520    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"]
Jul 16 21:54:04.924404 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:54:04.924378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc34873d_40f2_4153_a382_d69f6c703405.slice/crio-6d50c5dd3f61f91f34b9fda82c2178f6aff8f4ec6e798920dfc38c3f014faebc WatchSource:0}: Error finding container 6d50c5dd3f61f91f34b9fda82c2178f6aff8f4ec6e798920dfc38c3f014faebc: Status 404 returned error can't find the container with id 6d50c5dd3f61f91f34b9fda82c2178f6aff8f4ec6e798920dfc38c3f014faebc
Jul 16 21:54:05.765800 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:05.765771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:54:05.780425 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:05.780392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz" event={"ID":"dc34873d-40f2-4153-a382-d69f6c703405","Type":"ContainerStarted","Data":"6d50c5dd3f61f91f34b9fda82c2178f6aff8f4ec6e798920dfc38c3f014faebc"}
Jul 16 21:54:07.775922 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.775882    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"]
Jul 16 21:54:07.786667 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.786645    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"]
Jul 16 21:54:07.786823 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.786768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.790250 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.790228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.790831 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.790810    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v6ghg\""
Jul 16 21:54:07.790948 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.790861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 16 21:54:07.793881 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.793862    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 21:54:07.793993 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.793884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hhrcd\""
Jul 16 21:54:07.815391 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.815318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"]
Jul 16 21:54:07.825590 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.825568    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"]
Jul 16 21:54:07.862027 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.862001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4z9\" (UniqueName: \"kubernetes.io/projected/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-kube-api-access-8b4z9\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.862181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.862044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68pz\" (UniqueName: \"kubernetes.io/projected/f938169b-f9fb-4385-a27c-4e7450ed1366-kube-api-access-d68pz\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.862181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.862066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f938169b-f9fb-4385-a27c-4e7450ed1366-cert\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.862302 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.862208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.919075 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.919039    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:54:07.919172 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.919110    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:54:07.919172 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.919141    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:54:07.962855 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.962828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.962973 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.962875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4z9\" (UniqueName: \"kubernetes.io/projected/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-kube-api-access-8b4z9\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.962973 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.962905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68pz\" (UniqueName: \"kubernetes.io/projected/f938169b-f9fb-4385-a27c-4e7450ed1366-kube-api-access-d68pz\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.962973 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.962922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f938169b-f9fb-4385-a27c-4e7450ed1366-cert\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.965172 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.965144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f938169b-f9fb-4385-a27c-4e7450ed1366-cert\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:07.965380 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.965364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.977956 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.977932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4z9\" (UniqueName: \"kubernetes.io/projected/bb8353bb-5367-4dc1-8d85-3cead16b9d5b-kube-api-access-8b4z9\") pod \"llmisvc-controller-manager-777c6bd7d5-74qf2\" (UID: \"bb8353bb-5367-4dc1-8d85-3cead16b9d5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:07.978056 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:07.977992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68pz\" (UniqueName: \"kubernetes.io/projected/f938169b-f9fb-4385-a27c-4e7450ed1366-kube-api-access-d68pz\") pod \"kserve-controller-manager-d6f4b65b8-w5c8b\" (UID: \"f938169b-f9fb-4385-a27c-4e7450ed1366\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:08.101747 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.101649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:08.108389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.108369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:08.147201 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.147116    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:54:08.147424 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.147287    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerName="discovery" containerID="cri-o://aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f" gracePeriod=30
Jul 16 21:54:08.287803 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.287760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"]
Jul 16 21:54:08.292583 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:54:08.292536    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb8353bb_5367_4dc1_8d85_3cead16b9d5b.slice/crio-33af29416a434716216ce70af1a53599e2034cc8c8f6650cecb33bee06df2930 WatchSource:0}: Error finding container 33af29416a434716216ce70af1a53599e2034cc8c8f6650cecb33bee06df2930: Status 404 returned error can't find the container with id 33af29416a434716216ce70af1a53599e2034cc8c8f6650cecb33bee06df2930
Jul 16 21:54:08.336399 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.336375    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"]
Jul 16 21:54:08.339156 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:54:08.339123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf938169b_f9fb_4385_a27c_4e7450ed1366.slice/crio-2e8f3a0b40437eaef7db246e99ce2298f805d94a6a5d09cd275f030fbae5e4bf WatchSource:0}: Error finding container 2e8f3a0b40437eaef7db246e99ce2298f805d94a6a5d09cd275f030fbae5e4bf: Status 404 returned error can't find the container with id 2e8f3a0b40437eaef7db246e99ce2298f805d94a6a5d09cd275f030fbae5e4bf
Jul 16 21:54:08.396865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.396841    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:54:08.469017 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.468992    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469142 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469142 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469050    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469142 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dtzd\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469309 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469179    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469309 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469309 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig\") pod \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\" (UID: \"86f88e09-97c1-4f56-931e-9c7d7f770bbe\") "
Jul 16 21:54:08.469930 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.469903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:54:08.471830 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471784    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:54:08.471925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:54:08.471925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs" (OuterVolumeSpecName: "local-certs") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:54:08.471925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts" (OuterVolumeSpecName: "cacerts") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:54:08.471925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd" (OuterVolumeSpecName: "kube-api-access-8dtzd") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "kube-api-access-8dtzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:54:08.471925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.471888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token" (OuterVolumeSpecName: "istio-token") pod "86f88e09-97c1-4f56-931e-9c7d7f770bbe" (UID: "86f88e09-97c1-4f56-931e-9c7d7f770bbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:54:08.570178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570147    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-cacerts\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570174    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-dns-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570178 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570185    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570194    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtzd\" (UniqueName: \"kubernetes.io/projected/86f88e09-97c1-4f56-931e-9c7d7f770bbe-kube-api-access-8dtzd\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570204    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f88e09-97c1-4f56-931e-9c7d7f770bbe-local-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570221    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-csr-ca-configmap\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.570484 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.570232    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f88e09-97c1-4f56-931e-9c7d7f770bbe-istio-kubeconfig\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:08.797444 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.797409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz" event={"ID":"dc34873d-40f2-4153-a382-d69f6c703405","Type":"ContainerStarted","Data":"034175880c3535e011fbfd166570bf69334532dbaf9ef0d240f58161413d5355"}
Jul 16 21:54:08.798732 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.798703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2" event={"ID":"bb8353bb-5367-4dc1-8d85-3cead16b9d5b","Type":"ContainerStarted","Data":"33af29416a434716216ce70af1a53599e2034cc8c8f6650cecb33bee06df2930"}
Jul 16 21:54:08.799716 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.799694    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b" event={"ID":"f938169b-f9fb-4385-a27c-4e7450ed1366","Type":"ContainerStarted","Data":"2e8f3a0b40437eaef7db246e99ce2298f805d94a6a5d09cd275f030fbae5e4bf"}
Jul 16 21:54:08.800822 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.800805    2566 generic.go:356] "Generic (PLEG): container finished" podID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerID="aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f" exitCode=0
Jul 16 21:54:08.800885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.800846    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"
Jul 16 21:54:08.800885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.800870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" event={"ID":"86f88e09-97c1-4f56-931e-9c7d7f770bbe","Type":"ContainerDied","Data":"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"}
Jul 16 21:54:08.800950 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.800901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-57tdn" event={"ID":"86f88e09-97c1-4f56-931e-9c7d7f770bbe","Type":"ContainerDied","Data":"3baf28ef3e3f42e9669ac6f62810336461f77835fee670c065b839f967fd030c"}
Jul 16 21:54:08.800950 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.800921    2566 scope.go:117] "RemoveContainer" containerID="aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"
Jul 16 21:54:08.809970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.809938    2566 scope.go:117] "RemoveContainer" containerID="aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"
Jul 16 21:54:08.810180 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:54:08.810163    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f\": container with ID starting with aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f not found: ID does not exist" containerID="aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"
Jul 16 21:54:08.810241 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.810187    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f"} err="failed to get container status \"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f\": rpc error: code = NotFound desc = could not find container \"aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f\": container with ID starting with aa1483ea54d5dfa9f979a36e8a4cb414d4a044c4280024a79137753a377c311f not found: ID does not exist"
Jul 16 21:54:08.839289 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.839243    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz" podStartSLOduration=1.8468265480000001 podStartE2EDuration="4.839229416s" totalImagesPullingTime="2.992402868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:08.836960779 +0000 UTC m=+407.346827861" watchObservedRunningTime="2026-07-16 21:54:08.839229416 +0000 UTC m=+407.349096497"
Jul 16 21:54:08.862285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.862261    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:54:08.866677 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:08.866656    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-57tdn"]
Jul 16 21:54:09.717861 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:09.717828    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:09.723807 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:09.723780    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:09.809865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:09.809837    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:09.810697 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:09.810677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf47ptlz"
Jul 16 21:54:10.012821 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:10.012789    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" path="/var/lib/kubelet/pods/86f88e09-97c1-4f56-931e-9c7d7f770bbe/volumes"
Jul 16 21:54:10.755937 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:10.755900    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-nlx76"
Jul 16 21:54:11.822875 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:11.822828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2" event={"ID":"bb8353bb-5367-4dc1-8d85-3cead16b9d5b","Type":"ContainerStarted","Data":"bbcc0d52c5dca8e1af7e9fa5803163ddc8153e282a8c61e8a7fe2d4094b2189b"}
Jul 16 21:54:11.823347 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:11.822925    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:11.824881 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:11.824846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b" event={"ID":"f938169b-f9fb-4385-a27c-4e7450ed1366","Type":"ContainerStarted","Data":"45a77208178df3bbc5ce14bb8ecf065da3fe4f310b1f3804bb77e7b7ed0322f0"}
Jul 16 21:54:11.845545 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:11.845492    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2" podStartSLOduration=1.753242233 podStartE2EDuration="4.845472949s" totalImagesPullingTime="3.092230716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:11.843378975 +0000 UTC m=+410.353246057" watchObservedRunningTime="2026-07-16 21:54:11.845472949 +0000 UTC m=+410.355340022"
Jul 16 21:54:11.890716 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:11.890661    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b" podStartSLOduration=1.848761195 podStartE2EDuration="4.890645368s" totalImagesPullingTime="3.041884173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:11.88736569 +0000 UTC m=+410.397232772" watchObservedRunningTime="2026-07-16 21:54:11.890645368 +0000 UTC m=+410.400512450"
Jul 16 21:54:12.829347 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:12.829313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:42.831909 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:42.831878    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-74qf2"
Jul 16 21:54:43.839031 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:43.838998    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w5c8b"
Jul 16 21:54:55.050619 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.050578    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:54:55.051011 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.050947    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerName="discovery"
Jul 16 21:54:55.051011 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.050958    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerName="discovery"
Jul 16 21:54:55.051079 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.051023    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f88e09-97c1-4f56-931e-9c7d7f770bbe" containerName="discovery"
Jul 16 21:54:55.054408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.054385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:55.058493 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.058428    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:54:55.059555 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.059526    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-svkfl\""
Jul 16 21:54:55.059674 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.059558    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:54:55.068732 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.068696    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:54:55.073890 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.073853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t249p\" (UniqueName: \"kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p\") pod \"kuadrant-operator-catalog-vclxm\" (UID: \"aea22481-720c-4870-b113-58c406b0ad21\") " pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:55.174545 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.174496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t249p\" (UniqueName: \"kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p\") pod \"kuadrant-operator-catalog-vclxm\" (UID: \"aea22481-720c-4870-b113-58c406b0ad21\") " pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:55.189020 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.188989    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t249p\" (UniqueName: \"kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p\") pod \"kuadrant-operator-catalog-vclxm\" (UID: \"aea22481-720c-4870-b113-58c406b0ad21\") " pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:55.367534 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.367399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:55.391198 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.391152    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:54:55.510389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.510334    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:54:55.513751 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:54:55.513717    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea22481_720c_4870_b113_58c406b0ad21.slice/crio-51eebe584ade6436192a794a9eb0ccf7d0a9afd0787c455b7957ef87dede515c WatchSource:0}: Error finding container 51eebe584ade6436192a794a9eb0ccf7d0a9afd0787c455b7957ef87dede515c: Status 404 returned error can't find the container with id 51eebe584ade6436192a794a9eb0ccf7d0a9afd0787c455b7957ef87dede515c
Jul 16 21:54:55.586198 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.586157    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nv9jj"]
Jul 16 21:54:55.589099 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.589072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:54:55.599326 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.599290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nv9jj"]
Jul 16 21:54:55.678937 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.678826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpgw\" (UniqueName: \"kubernetes.io/projected/edfa8562-85ec-4ed4-911d-29ebc0e243da-kube-api-access-bkpgw\") pod \"kuadrant-operator-catalog-nv9jj\" (UID: \"edfa8562-85ec-4ed4-911d-29ebc0e243da\") " pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:54:55.780371 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.780323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpgw\" (UniqueName: \"kubernetes.io/projected/edfa8562-85ec-4ed4-911d-29ebc0e243da-kube-api-access-bkpgw\") pod \"kuadrant-operator-catalog-nv9jj\" (UID: \"edfa8562-85ec-4ed4-911d-29ebc0e243da\") " pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:54:55.788821 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.788790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpgw\" (UniqueName: \"kubernetes.io/projected/edfa8562-85ec-4ed4-911d-29ebc0e243da-kube-api-access-bkpgw\") pod \"kuadrant-operator-catalog-nv9jj\" (UID: \"edfa8562-85ec-4ed4-911d-29ebc0e243da\") " pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:54:55.900350 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:55.900305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:54:56.038640 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:56.038594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" event={"ID":"aea22481-720c-4870-b113-58c406b0ad21","Type":"ContainerStarted","Data":"51eebe584ade6436192a794a9eb0ccf7d0a9afd0787c455b7957ef87dede515c"}
Jul 16 21:54:56.054025 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:56.053995    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nv9jj"]
Jul 16 21:54:56.056208 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:54:56.056179    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfa8562_85ec_4ed4_911d_29ebc0e243da.slice/crio-2fd88cf296aa8c37e43aa8eb75715ae5f89471f3506d9fb4874cf91ee7fdf61a WatchSource:0}: Error finding container 2fd88cf296aa8c37e43aa8eb75715ae5f89471f3506d9fb4874cf91ee7fdf61a: Status 404 returned error can't find the container with id 2fd88cf296aa8c37e43aa8eb75715ae5f89471f3506d9fb4874cf91ee7fdf61a
Jul 16 21:54:57.047215 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:57.047166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj" event={"ID":"edfa8562-85ec-4ed4-911d-29ebc0e243da","Type":"ContainerStarted","Data":"2fd88cf296aa8c37e43aa8eb75715ae5f89471f3506d9fb4874cf91ee7fdf61a"}
Jul 16 21:54:58.053175 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.053132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" event={"ID":"aea22481-720c-4870-b113-58c406b0ad21","Type":"ContainerStarted","Data":"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"}
Jul 16 21:54:58.053674 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.053214    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" podUID="aea22481-720c-4870-b113-58c406b0ad21" containerName="registry-server" containerID="cri-o://455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b" gracePeriod=2
Jul 16 21:54:58.054957 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.054929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj" event={"ID":"edfa8562-85ec-4ed4-911d-29ebc0e243da","Type":"ContainerStarted","Data":"7ec9a6ec64c813a10d7f7ebf40efd021baaa269dbebcb9e8a2a00fbf699c671d"}
Jul 16 21:54:58.071497 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.071419    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" podStartSLOduration=0.976537796 podStartE2EDuration="3.071401618s" totalImagesPullingTime="2.094863822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:58.070246839 +0000 UTC m=+456.580113921" watchObservedRunningTime="2026-07-16 21:54:58.071401618 +0000 UTC m=+456.581268700"
Jul 16 21:54:58.089974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.089908    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj" podStartSLOduration=1.536818727 podStartE2EDuration="3.08988591s" totalImagesPullingTime="1.553067183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:58.088449884 +0000 UTC m=+456.598316967" watchObservedRunningTime="2026-07-16 21:54:58.08988591 +0000 UTC m=+456.599753026"
Jul 16 21:54:58.310716 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.310633    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:58.406184 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.406139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t249p\" (UniqueName: \"kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p\") pod \"aea22481-720c-4870-b113-58c406b0ad21\" (UID: \"aea22481-720c-4870-b113-58c406b0ad21\") "
Jul 16 21:54:58.408412 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.408382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p" (OuterVolumeSpecName: "kube-api-access-t249p") pod "aea22481-720c-4870-b113-58c406b0ad21" (UID: "aea22481-720c-4870-b113-58c406b0ad21"). InnerVolumeSpecName "kube-api-access-t249p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:54:58.506920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:58.506877    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t249p\" (UniqueName: \"kubernetes.io/projected/aea22481-720c-4870-b113-58c406b0ad21-kube-api-access-t249p\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:54:59.061281 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.061244    2566 generic.go:356] "Generic (PLEG): container finished" podID="aea22481-720c-4870-b113-58c406b0ad21" containerID="455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b" exitCode=0
Jul 16 21:54:59.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.061289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vclxm"
Jul 16 21:54:59.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.061331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" event={"ID":"aea22481-720c-4870-b113-58c406b0ad21","Type":"ContainerDied","Data":"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"}
Jul 16 21:54:59.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.061380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vclxm" event={"ID":"aea22481-720c-4870-b113-58c406b0ad21","Type":"ContainerDied","Data":"51eebe584ade6436192a794a9eb0ccf7d0a9afd0787c455b7957ef87dede515c"}
Jul 16 21:54:59.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.061402    2566 scope.go:117] "RemoveContainer" containerID="455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"
Jul 16 21:54:59.072265 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.072241    2566 scope.go:117] "RemoveContainer" containerID="455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"
Jul 16 21:54:59.072632 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:54:59.072608    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b\": container with ID starting with 455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b not found: ID does not exist" containerID="455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"
Jul 16 21:54:59.072692 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.072643    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b"} err="failed to get container status \"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b\": rpc error: code = NotFound desc = could not find container \"455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b\": container with ID starting with 455a67e4926480b78f22caf52ef57c8e1ec159b2901ccc7445cb3a6263a83f0b not found: ID does not exist"
Jul 16 21:54:59.083783 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.083742    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:54:59.087936 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:54:59.087896    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vclxm"]
Jul 16 21:55:00.011238 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:00.011195    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea22481-720c-4870-b113-58c406b0ad21" path="/var/lib/kubelet/pods/aea22481-720c-4870-b113-58c406b0ad21/volumes"
Jul 16 21:55:05.900666 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:05.900619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:55:05.901102 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:05.900696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:55:05.924370 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:05.924330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:55:06.119629 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:06.119599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-nv9jj"
Jul 16 21:55:24.086326 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.086280    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-648pc"]
Jul 16 21:55:24.086780 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.086670    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea22481-720c-4870-b113-58c406b0ad21" containerName="registry-server"
Jul 16 21:55:24.086780 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.086684    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea22481-720c-4870-b113-58c406b0ad21" containerName="registry-server"
Jul 16 21:55:24.086780 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.086756    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea22481-720c-4870-b113-58c406b0ad21" containerName="registry-server"
Jul 16 21:55:24.090948 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.090920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:24.097684 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.097651    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4jddl\""
Jul 16 21:55:24.128752 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.128718    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-648pc"]
Jul 16 21:55:24.241755 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.241713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmqt\" (UniqueName: \"kubernetes.io/projected/0407b5d5-b43d-4dfc-8901-c1b0a68a9c68-kube-api-access-fsmqt\") pod \"authorino-operator-75cc8d5cb7-648pc\" (UID: \"0407b5d5-b43d-4dfc-8901-c1b0a68a9c68\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:24.343227 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.343130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmqt\" (UniqueName: \"kubernetes.io/projected/0407b5d5-b43d-4dfc-8901-c1b0a68a9c68-kube-api-access-fsmqt\") pod \"authorino-operator-75cc8d5cb7-648pc\" (UID: \"0407b5d5-b43d-4dfc-8901-c1b0a68a9c68\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:24.372129 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.372094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmqt\" (UniqueName: \"kubernetes.io/projected/0407b5d5-b43d-4dfc-8901-c1b0a68a9c68-kube-api-access-fsmqt\") pod \"authorino-operator-75cc8d5cb7-648pc\" (UID: \"0407b5d5-b43d-4dfc-8901-c1b0a68a9c68\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:24.401898 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.401858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:24.571898 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.571863    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-648pc"]
Jul 16 21:55:24.575578 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:24.575540    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0407b5d5_b43d_4dfc_8901_c1b0a68a9c68.slice/crio-00665067e8128f9fb9999fb980c648d639b131be20f765488d405b69741e50da WatchSource:0}: Error finding container 00665067e8128f9fb9999fb980c648d639b131be20f765488d405b69741e50da: Status 404 returned error can't find the container with id 00665067e8128f9fb9999fb980c648d639b131be20f765488d405b69741e50da
Jul 16 21:55:24.930591 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.930549    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7755fb5649-bjhcn"]
Jul 16 21:55:24.935129 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.935099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:24.983371 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:24.983330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7755fb5649-bjhcn"]
Jul 16 21:55:25.049827 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-trusted-ca-bundle\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-service-ca\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-oauth-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-oauth-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050010 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.049928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.050171 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.050015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs97r\" (UniqueName: \"kubernetes.io/projected/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-kube-api-access-qs97r\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151389 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-trusted-ca-bundle\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-service-ca\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-oauth-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-oauth-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.151889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.151607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs97r\" (UniqueName: \"kubernetes.io/projected/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-kube-api-access-qs97r\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.152286 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.152253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-service-ca\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.152375 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.152313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.152531 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.152429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-trusted-ca-bundle\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.152531 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.152428    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-oauth-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.154350 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.154307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-oauth-config\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.154465 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.154319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-console-serving-cert\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.189584 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.189480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc" event={"ID":"0407b5d5-b43d-4dfc-8901-c1b0a68a9c68","Type":"ContainerStarted","Data":"00665067e8128f9fb9999fb980c648d639b131be20f765488d405b69741e50da"}
Jul 16 21:55:25.235011 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.234978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs97r\" (UniqueName: \"kubernetes.io/projected/b51aa90f-f006-4eb8-aea6-a2a8f5afb138-kube-api-access-qs97r\") pod \"console-7755fb5649-bjhcn\" (UID: \"b51aa90f-f006-4eb8-aea6-a2a8f5afb138\") " pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.246435 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.246392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:25.506759 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:25.506719    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7755fb5649-bjhcn"]
Jul 16 21:55:25.508755 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:25.508725    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51aa90f_f006_4eb8_aea6_a2a8f5afb138.slice/crio-b03d14d093fc5e1eef37b67645b719813baee816487a4c5c8d2b6f1fe9fb1681 WatchSource:0}: Error finding container b03d14d093fc5e1eef37b67645b719813baee816487a4c5c8d2b6f1fe9fb1681: Status 404 returned error can't find the container with id b03d14d093fc5e1eef37b67645b719813baee816487a4c5c8d2b6f1fe9fb1681
Jul 16 21:55:26.197970 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:26.197927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7755fb5649-bjhcn" event={"ID":"b51aa90f-f006-4eb8-aea6-a2a8f5afb138","Type":"ContainerStarted","Data":"f412717a5347779a29b60d83bb5c999f063ebb6407712711f7d36a7707fbbc37"}
Jul 16 21:55:26.198439 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:26.197992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7755fb5649-bjhcn" event={"ID":"b51aa90f-f006-4eb8-aea6-a2a8f5afb138","Type":"ContainerStarted","Data":"b03d14d093fc5e1eef37b67645b719813baee816487a4c5c8d2b6f1fe9fb1681"}
Jul 16 21:55:26.245660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:26.245596    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7755fb5649-bjhcn" podStartSLOduration=2.245579406 podStartE2EDuration="2.245579406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:26.244803588 +0000 UTC m=+484.754670674" watchObservedRunningTime="2026-07-16 21:55:26.245579406 +0000 UTC m=+484.755446489"
Jul 16 21:55:27.204987 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:27.204894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc" event={"ID":"0407b5d5-b43d-4dfc-8901-c1b0a68a9c68","Type":"ContainerStarted","Data":"12b134d49ff3c847a55deabc5aae584b407764d134c969e1e232f9cfd30a79fa"}
Jul 16 21:55:27.205365 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:27.205207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:27.302161 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:27.302108    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc" podStartSLOduration=0.997235925 podStartE2EDuration="3.302088899s" totalImagesPullingTime="2.304852974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:27.301601257 +0000 UTC m=+485.811468341" watchObservedRunningTime="2026-07-16 21:55:27.302088899 +0000 UTC m=+485.811955980"
Jul 16 21:55:29.526750 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.526704    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"]
Jul 16 21:55:29.530119 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.530095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.535342 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.535314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wqpb2\""
Jul 16 21:55:29.611378 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.611335    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"]
Jul 16 21:55:29.690789 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.690743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.690974 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.690834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52c9\" (UniqueName: \"kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.791823 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.791726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.791977 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.791871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52c9\" (UniqueName: \"kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.792171 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.792148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.824018 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.823985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52c9\" (UniqueName: \"kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:29.841869 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:29.841828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:30.020172 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:30.020129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"]
Jul 16 21:55:30.022646 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:30.022610    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f2ba58_58a1_4423_b491_55d983bc0491.slice/crio-21bfa7ac34495705e89d54890f04dfe85e73fca75265b36fd2a1baaf72c00211 WatchSource:0}: Error finding container 21bfa7ac34495705e89d54890f04dfe85e73fca75265b36fd2a1baaf72c00211: Status 404 returned error can't find the container with id 21bfa7ac34495705e89d54890f04dfe85e73fca75265b36fd2a1baaf72c00211
Jul 16 21:55:30.222176 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:30.222081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4" event={"ID":"a4f2ba58-58a1-4423-b491-55d983bc0491","Type":"ContainerStarted","Data":"21bfa7ac34495705e89d54890f04dfe85e73fca75265b36fd2a1baaf72c00211"}
Jul 16 21:55:31.534645 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.534599    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"]
Jul 16 21:55:31.540259 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.540228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:31.544448 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.544422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-99lm9\""
Jul 16 21:55:31.569173 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.569137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"]
Jul 16 21:55:31.709840 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.709789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnk7\" (UniqueName: \"kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7\") pod \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" (UID: \"e3204c76-3978-406f-b9f4-691f82034309\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:31.810788 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.810689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnk7\" (UniqueName: \"kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7\") pod \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" (UID: \"e3204c76-3978-406f-b9f4-691f82034309\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:31.848590 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.848546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnk7\" (UniqueName: \"kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7\") pod \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" (UID: \"e3204c76-3978-406f-b9f4-691f82034309\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:31.853333 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:31.853304    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:32.036643 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:32.036603    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"]
Jul 16 21:55:32.040595 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:32.040553    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3204c76_3978_406f_b9f4_691f82034309.slice/crio-2207a8daa1e030ce57ff11cac22e3ae3515a066b5cbe78921a78a304286c7fd0 WatchSource:0}: Error finding container 2207a8daa1e030ce57ff11cac22e3ae3515a066b5cbe78921a78a304286c7fd0: Status 404 returned error can't find the container with id 2207a8daa1e030ce57ff11cac22e3ae3515a066b5cbe78921a78a304286c7fd0
Jul 16 21:55:32.234699 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:32.234597    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2" event={"ID":"e3204c76-3978-406f-b9f4-691f82034309","Type":"ContainerStarted","Data":"2207a8daa1e030ce57ff11cac22e3ae3515a066b5cbe78921a78a304286c7fd0"}
Jul 16 21:55:34.248924 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.248869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2" event={"ID":"e3204c76-3978-406f-b9f4-691f82034309","Type":"ContainerStarted","Data":"7e8add51b246a0faf5ccd09fade3a93e753faae3bfd499ca81a68a64ae27e51f"}
Jul 16 21:55:34.249358 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.248979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:34.327915 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.327852    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2" podStartSLOduration=1.28654168 podStartE2EDuration="3.32783347s" totalImagesPullingTime="2.04129179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:34.325106503 +0000 UTC m=+492.834973586" watchObservedRunningTime="2026-07-16 21:55:34.32783347 +0000 UTC m=+492.837700551"
Jul 16 21:55:34.536850 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.536807    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"]
Jul 16 21:55:34.540538 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.540508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:34.544023 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.543988    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vpn86\""
Jul 16 21:55:34.544233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.544214    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 16 21:55:34.566870 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.566831    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"]
Jul 16 21:55:34.637524 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.637485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stsps\" (UniqueName: \"kubernetes.io/projected/bb659723-9fd2-4429-995d-693d040a8151-kube-api-access-stsps\") pod \"dns-operator-controller-manager-864dd78c4c-mjwvd\" (UID: \"bb659723-9fd2-4429-995d-693d040a8151\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:34.738513 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.738446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stsps\" (UniqueName: \"kubernetes.io/projected/bb659723-9fd2-4429-995d-693d040a8151-kube-api-access-stsps\") pod \"dns-operator-controller-manager-864dd78c4c-mjwvd\" (UID: \"bb659723-9fd2-4429-995d-693d040a8151\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:34.763510 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.763475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stsps\" (UniqueName: \"kubernetes.io/projected/bb659723-9fd2-4429-995d-693d040a8151-kube-api-access-stsps\") pod \"dns-operator-controller-manager-864dd78c4c-mjwvd\" (UID: \"bb659723-9fd2-4429-995d-693d040a8151\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:34.852665 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:34.852565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:35.247078 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:35.247033    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:35.247292 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:35.247102    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:35.253681 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:35.253645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:35.259410 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:35.259375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7755fb5649-bjhcn"
Jul 16 21:55:35.406051 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:35.405997    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:55:36.032840 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:36.032805    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"]
Jul 16 21:55:36.037102 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:36.037068    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb659723_9fd2_4429_995d_693d040a8151.slice/crio-e92902fecc441d545c281608498431080fba55ea6acba624d210d0206cbed297 WatchSource:0}: Error finding container e92902fecc441d545c281608498431080fba55ea6acba624d210d0206cbed297: Status 404 returned error can't find the container with id e92902fecc441d545c281608498431080fba55ea6acba624d210d0206cbed297
Jul 16 21:55:36.261801 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:36.261759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4" event={"ID":"a4f2ba58-58a1-4423-b491-55d983bc0491","Type":"ContainerStarted","Data":"99cf781cd2f0045303d1d93e99de7c1aaeeedfeb4bacd896876381f93b0c65ff"}
Jul 16 21:55:36.262218 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:36.261870    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:36.263233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:36.263197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd" event={"ID":"bb659723-9fd2-4429-995d-693d040a8151","Type":"ContainerStarted","Data":"e92902fecc441d545c281608498431080fba55ea6acba624d210d0206cbed297"}
Jul 16 21:55:36.314741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:36.314676    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4" podStartSLOduration=1.413965723 podStartE2EDuration="7.314655564s" totalImagesPullingTime="5.900689841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:36.312285809 +0000 UTC m=+494.822152893" watchObservedRunningTime="2026-07-16 21:55:36.314655564 +0000 UTC m=+494.824522645"
Jul 16 21:55:38.213781 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:38.213691    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-648pc"
Jul 16 21:55:38.277420 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:38.277276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd" event={"ID":"bb659723-9fd2-4429-995d-693d040a8151","Type":"ContainerStarted","Data":"4ce8670c868fe330900520d030ec06cefcc01b7198d26df3e03383a124e02293"}
Jul 16 21:55:38.277604 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:38.277495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:38.317574 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:38.317514    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd" podStartSLOduration=2.543461971 podStartE2EDuration="4.317495255s" totalImagesPullingTime="1.774033284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:38.314426244 +0000 UTC m=+496.824293326" watchObservedRunningTime="2026-07-16 21:55:38.317495255 +0000 UTC m=+496.827362340"
Jul 16 21:55:39.574163 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.574127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"]
Jul 16 21:55:39.577859 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.577837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.583639 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.583604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbtm\" (UniqueName: \"kubernetes.io/projected/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-kube-api-access-cfbtm\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.583804 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.583762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.583858 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.583825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.586659 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.586634    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 21:55:39.586901 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.586885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rbwj4\""
Jul 16 21:55:39.587078 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.587061    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 21:55:39.606278 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.606241    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"]
Jul 16 21:55:39.684848 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.684811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.685056 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.684861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.685056 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.684889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbtm\" (UniqueName: \"kubernetes.io/projected/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-kube-api-access-cfbtm\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.685640 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.685613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.687285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.687258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.723505 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.723473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbtm\" (UniqueName: \"kubernetes.io/projected/1ddb94a5-fff7-46e9-bdcc-22818765e0b2-kube-api-access-cfbtm\") pod \"kuadrant-console-plugin-66b5774985-qlxl2\" (UID: \"1ddb94a5-fff7-46e9-bdcc-22818765e0b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:39.887065 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:39.886965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"
Jul 16 21:55:40.055242 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:55:40.055207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ddb94a5_fff7_46e9_bdcc_22818765e0b2.slice/crio-9b598703a03fa4b9ae72ff4f8c28f28a8e6064daf5759210290437f22329ce3a WatchSource:0}: Error finding container 9b598703a03fa4b9ae72ff4f8c28f28a8e6064daf5759210290437f22329ce3a: Status 404 returned error can't find the container with id 9b598703a03fa4b9ae72ff4f8c28f28a8e6064daf5759210290437f22329ce3a
Jul 16 21:55:40.055242 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:40.055227    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2"]
Jul 16 21:55:40.288239 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:40.288199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2" event={"ID":"1ddb94a5-fff7-46e9-bdcc-22818765e0b2","Type":"ContainerStarted","Data":"9b598703a03fa4b9ae72ff4f8c28f28a8e6064daf5759210290437f22329ce3a"}
Jul 16 21:55:45.257377 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:45.257337    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:47.272743 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:47.272706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:49.285181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:49.285145    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-mjwvd"
Jul 16 21:55:50.191277 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.191244    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"]
Jul 16 21:55:50.191517 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.191466    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" containerName="manager" containerID="cri-o://99cf781cd2f0045303d1d93e99de7c1aaeeedfeb4bacd896876381f93b0c65ff" gracePeriod=2
Jul 16 21:55:50.218584 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.218545    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"]
Jul 16 21:55:50.241853 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.241811    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"]
Jul 16 21:55:50.242121 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.242089    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2" podUID="e3204c76-3978-406f-b9f4-691f82034309" containerName="manager" containerID="cri-o://7e8add51b246a0faf5ccd09fade3a93e753faae3bfd499ca81a68a64ae27e51f" gracePeriod=2
Jul 16 21:55:50.263657 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.263560    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"]
Jul 16 21:55:50.270609 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.270556    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e3204c76-3978-406f-b9f4-691f82034309\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-7s8x2\": pods \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:50.274557 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.274507    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:50.333861 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.333813    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:55:50.334290 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334260    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" containerName="manager"
Jul 16 21:55:50.334290 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334275    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" containerName="manager"
Jul 16 21:55:50.334408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334301    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3204c76-3978-406f-b9f4-691f82034309" containerName="manager"
Jul 16 21:55:50.334408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334306    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3204c76-3978-406f-b9f4-691f82034309" containerName="manager"
Jul 16 21:55:50.334408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334394    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3204c76-3978-406f-b9f4-691f82034309" containerName="manager"
Jul 16 21:55:50.334539 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.334412    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" containerName="manager"
Jul 16 21:55:50.341149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.341121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.376112 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.375975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"]
Jul 16 21:55:50.380830 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.380776    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:50.384092 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.384064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:55:50.390520 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.390489    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4lh\" (UniqueName: \"kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.390697 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.390599    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.413270 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.413225    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:55:50.469282 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.469054    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="e3204c76-3978-406f-b9f4-691f82034309" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:50.471050 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.471020    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"]
Jul 16 21:55:50.472172 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.472137    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:55:50.491847 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.491803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.492054 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.491905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9shg\" (UniqueName: \"kubernetes.io/projected/1639d5a9-19bf-433d-a250-02c54261fc78-kube-api-access-n9shg\") pod \"limitador-operator-controller-manager-859d7b78bd-kt4t9\" (UID: \"1639d5a9-19bf-433d-a250-02c54261fc78\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:55:50.492054 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.491992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4lh\" (UniqueName: \"kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.492283 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.492247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.511049 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.511016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4lh\" (UniqueName: \"kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-94tft\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.531803 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:50.531766    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="e3204c76-3978-406f-b9f4-691f82034309" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:50.592838 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.592800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9shg\" (UniqueName: \"kubernetes.io/projected/1639d5a9-19bf-433d-a250-02c54261fc78-kube-api-access-n9shg\") pod \"limitador-operator-controller-manager-859d7b78bd-kt4t9\" (UID: \"1639d5a9-19bf-433d-a250-02c54261fc78\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:55:50.656357 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.656315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:55:50.662693 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.662650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9shg\" (UniqueName: \"kubernetes.io/projected/1639d5a9-19bf-433d-a250-02c54261fc78-kube-api-access-n9shg\") pod \"limitador-operator-controller-manager-859d7b78bd-kt4t9\" (UID: \"1639d5a9-19bf-433d-a250-02c54261fc78\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:55:50.696712 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:55:50.696666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:55:52.009325 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:52.009282    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7s8x2\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="e3204c76-3978-406f-b9f4-691f82034309" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:55:52.018285 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:55:52.017969    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2jkx4\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:56:00.433005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:00.432963    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f8f99b575-dnkb6" podUID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" containerName="console" containerID="cri-o://dda971a3e32b53dc27afb4d42a3192527daee5e34f542fc0ec9451eb513fd6da" gracePeriod=15
Jul 16 21:56:01.429408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.427066    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4f2ba58-58a1-4423-b491-55d983bc0491" containerID="99cf781cd2f0045303d1d93e99de7c1aaeeedfeb4bacd896876381f93b0c65ff" exitCode=0
Jul 16 21:56:01.429408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.429036    2566 generic.go:356] "Generic (PLEG): container finished" podID="e3204c76-3978-406f-b9f4-691f82034309" containerID="7e8add51b246a0faf5ccd09fade3a93e753faae3bfd499ca81a68a64ae27e51f" exitCode=0
Jul 16 21:56:01.432805 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.432669    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" containerID="dda971a3e32b53dc27afb4d42a3192527daee5e34f542fc0ec9451eb513fd6da" exitCode=0
Jul 16 21:56:01.432805 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.432735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8f99b575-dnkb6" event={"ID":"8f55b345-edce-47dd-933f-ab2b7b5e7a7b","Type":"ContainerDied","Data":"dda971a3e32b53dc27afb4d42a3192527daee5e34f542fc0ec9451eb513fd6da"}
Jul 16 21:56:01.476278 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.476248    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:56:01.482262 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.482231    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:56:01.492275 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.492215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:56:01.606878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.606841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume\") pod \"a4f2ba58-58a1-4423-b491-55d983bc0491\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") "
Jul 16 21:56:01.606878 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.606883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w52c9\" (UniqueName: \"kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9\") pod \"a4f2ba58-58a1-4423-b491-55d983bc0491\" (UID: \"a4f2ba58-58a1-4423-b491-55d983bc0491\") "
Jul 16 21:56:01.607117 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.606912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607117 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.606946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8gvz\" (UniqueName: \"kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607117 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607058    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607263 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfnk7\" (UniqueName: \"kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7\") pod \"e3204c76-3978-406f-b9f4-691f82034309\" (UID: \"e3204c76-3978-406f-b9f4-691f82034309\") "
Jul 16 21:56:01.607411 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607411 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert\") pod \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\" (UID: \"8f55b345-edce-47dd-933f-ab2b7b5e7a7b\") "
Jul 16 21:56:01.607544 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config" (OuterVolumeSpecName: "console-config") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:01.607544 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4f2ba58-58a1-4423-b491-55d983bc0491" (UID: "a4f2ba58-58a1-4423-b491-55d983bc0491"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:01.607544 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607530    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:01.607706 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:01.607762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607714    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4f2ba58-58a1-4423-b491-55d983bc0491-extensions-socket-volume\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.607762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607736    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.607762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607750    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.607921 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607763    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-oauth-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.607979 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.607956    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:01.609287 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.609262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9" (OuterVolumeSpecName: "kube-api-access-w52c9") pod "a4f2ba58-58a1-4423-b491-55d983bc0491" (UID: "a4f2ba58-58a1-4423-b491-55d983bc0491"). InnerVolumeSpecName "kube-api-access-w52c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:01.609400 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.609385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:56:01.609645 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.609614    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:56:01.609719 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.609692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz" (OuterVolumeSpecName: "kube-api-access-v8gvz") pod "8f55b345-edce-47dd-933f-ab2b7b5e7a7b" (UID: "8f55b345-edce-47dd-933f-ab2b7b5e7a7b"). InnerVolumeSpecName "kube-api-access-v8gvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:01.609867 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.609851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7" (OuterVolumeSpecName: "kube-api-access-rfnk7") pod "e3204c76-3978-406f-b9f4-691f82034309" (UID: "e3204c76-3978-406f-b9f4-691f82034309"). InnerVolumeSpecName "kube-api-access-rfnk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:01.708470 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708412    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnk7\" (UniqueName: \"kubernetes.io/projected/e3204c76-3978-406f-b9f4-691f82034309-kube-api-access-rfnk7\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.708622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708483    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-oauth-config\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.708622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708495    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-console-serving-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.708622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708504    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w52c9\" (UniqueName: \"kubernetes.io/projected/a4f2ba58-58a1-4423-b491-55d983bc0491-kube-api-access-w52c9\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.708622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708512    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-trusted-ca-bundle\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.708622 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.708521    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gvz\" (UniqueName: \"kubernetes.io/projected/8f55b345-edce-47dd-933f-ab2b7b5e7a7b-kube-api-access-v8gvz\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:01.714831 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.714797    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:56:01.721873 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:01.721850    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"]
Jul 16 21:56:01.723750 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:01.723723    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef3fed1_a0f0_4301_b60c_4ef3562114c0.slice/crio-ef66b22b49d95e7c0ae21c3fcec59c8f665e947f27e9d994e214fb2b8d3780cf WatchSource:0}: Error finding container ef66b22b49d95e7c0ae21c3fcec59c8f665e947f27e9d994e214fb2b8d3780cf: Status 404 returned error can't find the container with id ef66b22b49d95e7c0ae21c3fcec59c8f665e947f27e9d994e214fb2b8d3780cf
Jul 16 21:56:01.724637 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:01.724618    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1639d5a9_19bf_433d_a250_02c54261fc78.slice/crio-6ae1a1f3fe5618ee65ac40f002a2b69c4c0df1a0e6a003e014f60e82841af10c WatchSource:0}: Error finding container 6ae1a1f3fe5618ee65ac40f002a2b69c4c0df1a0e6a003e014f60e82841af10c: Status 404 returned error can't find the container with id 6ae1a1f3fe5618ee65ac40f002a2b69c4c0df1a0e6a003e014f60e82841af10c
Jul 16 21:56:02.011555 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.011474    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f2ba58-58a1-4423-b491-55d983bc0491" path="/var/lib/kubelet/pods/a4f2ba58-58a1-4423-b491-55d983bc0491/volumes"
Jul 16 21:56:02.011805 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.011791    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3204c76-3978-406f-b9f4-691f82034309" path="/var/lib/kubelet/pods/e3204c76-3978-406f-b9f4-691f82034309/volumes"
Jul 16 21:56:02.439324 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.439216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9" event={"ID":"1639d5a9-19bf-433d-a250-02c54261fc78","Type":"ContainerStarted","Data":"c5b277b3834f30d436c654eea4f74f28d809c154648a230c1f38d993fd2b617d"}
Jul 16 21:56:02.439518 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.439332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9" event={"ID":"1639d5a9-19bf-433d-a250-02c54261fc78","Type":"ContainerStarted","Data":"6ae1a1f3fe5618ee65ac40f002a2b69c4c0df1a0e6a003e014f60e82841af10c"}
Jul 16 21:56:02.439518 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.439367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:56:02.440774 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.440749    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2jkx4"
Jul 16 21:56:02.440913 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.440747    2566 scope.go:117] "RemoveContainer" containerID="99cf781cd2f0045303d1d93e99de7c1aaeeedfeb4bacd896876381f93b0c65ff"
Jul 16 21:56:02.443110 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.442846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2" event={"ID":"1ddb94a5-fff7-46e9-bdcc-22818765e0b2","Type":"ContainerStarted","Data":"5afd981bdb985322b6ff12a6151d688fd14e18b058ee7ac0daddb10db776eb36"}
Jul 16 21:56:02.444246 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.444224    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7s8x2"
Jul 16 21:56:02.445564 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.445541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8f99b575-dnkb6" event={"ID":"8f55b345-edce-47dd-933f-ab2b7b5e7a7b","Type":"ContainerDied","Data":"527b320cff95bb39141bdd2caeccd507ee3ee0c4e39dc387b32c9ff1ac37adc0"}
Jul 16 21:56:02.445802 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.445550    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8f99b575-dnkb6"
Jul 16 21:56:02.447321 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.447293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" event={"ID":"bef3fed1-a0f0-4301-b60c-4ef3562114c0","Type":"ContainerStarted","Data":"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"}
Jul 16 21:56:02.447415 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.447343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" event={"ID":"bef3fed1-a0f0-4301-b60c-4ef3562114c0","Type":"ContainerStarted","Data":"ef66b22b49d95e7c0ae21c3fcec59c8f665e947f27e9d994e214fb2b8d3780cf"}
Jul 16 21:56:02.447500 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.447435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:56:02.451899 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.451879    2566 scope.go:117] "RemoveContainer" containerID="7e8add51b246a0faf5ccd09fade3a93e753faae3bfd499ca81a68a64ae27e51f"
Jul 16 21:56:02.460080 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.459966    2566 scope.go:117] "RemoveContainer" containerID="dda971a3e32b53dc27afb4d42a3192527daee5e34f542fc0ec9451eb513fd6da"
Jul 16 21:56:02.482671 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.482619    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9" podStartSLOduration=12.482603165 podStartE2EDuration="12.482603165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:02.480359722 +0000 UTC m=+520.990226815" watchObservedRunningTime="2026-07-16 21:56:02.482603165 +0000 UTC m=+520.992470225"
Jul 16 21:56:02.511306 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.511272    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:56:02.528215 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.528184    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f8f99b575-dnkb6"]
Jul 16 21:56:02.590617 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.590566    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlxl2" podStartSLOduration=2.282352002 podStartE2EDuration="23.590550112s" totalImagesPullingTime="21.30819811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:02.58536965 +0000 UTC m=+521.095236743" watchObservedRunningTime="2026-07-16 21:56:02.590550112 +0000 UTC m=+521.100417193"
Jul 16 21:56:02.623109 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:02.623036    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" podStartSLOduration=12.623011186 podStartE2EDuration="12.623011186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:02.620806391 +0000 UTC m=+521.130673474" watchObservedRunningTime="2026-07-16 21:56:02.623011186 +0000 UTC m=+521.132878270"
Jul 16 21:56:04.011628 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:04.011597    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" path="/var/lib/kubelet/pods/8f55b345-edce-47dd-933f-ab2b7b5e7a7b/volumes"
Jul 16 21:56:10.583735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.583691    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:56:10.584153 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.583906    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" podUID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" containerName="manager" containerID="cri-o://7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714" gracePeriod=10
Jul 16 21:56:10.585819 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.585793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:56:10.844771 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.844707    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:56:10.993895 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.993853    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp4lh\" (UniqueName: \"kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh\") pod \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") "
Jul 16 21:56:10.994070 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.993938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume\") pod \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\" (UID: \"bef3fed1-a0f0-4301-b60c-4ef3562114c0\") "
Jul 16 21:56:10.994379 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.994343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bef3fed1-a0f0-4301-b60c-4ef3562114c0" (UID: "bef3fed1-a0f0-4301-b60c-4ef3562114c0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:56:10.996070 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:10.996041    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh" (OuterVolumeSpecName: "kube-api-access-tp4lh") pod "bef3fed1-a0f0-4301-b60c-4ef3562114c0" (UID: "bef3fed1-a0f0-4301-b60c-4ef3562114c0"). InnerVolumeSpecName "kube-api-access-tp4lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:11.095287 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.095195    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4lh\" (UniqueName: \"kubernetes.io/projected/bef3fed1-a0f0-4301-b60c-4ef3562114c0-kube-api-access-tp4lh\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:11.095287 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.095233    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bef3fed1-a0f0-4301-b60c-4ef3562114c0-extensions-socket-volume\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:11.498864 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.498830    2566 generic.go:356] "Generic (PLEG): container finished" podID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" containerID="7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714" exitCode=0
Jul 16 21:56:11.499018 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.498874    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"
Jul 16 21:56:11.499018 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.498882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" event={"ID":"bef3fed1-a0f0-4301-b60c-4ef3562114c0","Type":"ContainerDied","Data":"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"}
Jul 16 21:56:11.499116 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.499020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft" event={"ID":"bef3fed1-a0f0-4301-b60c-4ef3562114c0","Type":"ContainerDied","Data":"ef66b22b49d95e7c0ae21c3fcec59c8f665e947f27e9d994e214fb2b8d3780cf"}
Jul 16 21:56:11.499116 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.499062    2566 scope.go:117] "RemoveContainer" containerID="7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"
Jul 16 21:56:11.508515 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.508496    2566 scope.go:117] "RemoveContainer" containerID="7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"
Jul 16 21:56:11.508809 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:56:11.508786    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714\": container with ID starting with 7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714 not found: ID does not exist" containerID="7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"
Jul 16 21:56:11.508860 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.508818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714"} err="failed to get container status \"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714\": rpc error: code = NotFound desc = could not find container \"7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714\": container with ID starting with 7e6cf721c7f6cdb32998807854ee3f663367436a4c849481d68c1b0c721c7714 not found: ID does not exist"
Jul 16 21:56:11.536721 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.536688    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:56:11.553584 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:11.553543    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-94tft"]
Jul 16 21:56:12.012009 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:12.011973    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" path="/var/lib/kubelet/pods/bef3fed1-a0f0-4301-b60c-4ef3562114c0/volumes"
Jul 16 21:56:13.457418 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:13.457390    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt4t9"
Jul 16 21:56:27.200766 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.200732    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"]
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201108    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" containerName="console"
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201118    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" containerName="console"
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201136    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" containerName="manager"
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" containerName="manager"
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201194    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef3fed1-a0f0-4301-b60c-4ef3562114c0" containerName="manager"
Jul 16 21:56:27.201249 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.201204    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f55b345-edce-47dd-933f-ab2b7b5e7a7b" containerName="console"
Jul 16 21:56:27.260099 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.260060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"]
Jul 16 21:56:27.260259 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.260175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.263576 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.263550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2ht77\""
Jul 16 21:56:27.344749 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwj6\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-kube-api-access-vzwj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.344929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456b15ec-9505-47ae-97a2-ec32e7c811d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.344929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.344929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.344929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.344929 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.345126 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.345126 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.345126 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.344999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446266 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446266 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446568 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwj6\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-kube-api-access-vzwj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446873 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456b15ec-9505-47ae-97a2-ec32e7c811d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446873 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.446873 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.447313 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.446956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.447313 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.447085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.447313 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.447267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456b15ec-9505-47ae-97a2-ec32e7c811d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.448728 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.448709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.448945 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.448920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.457653 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.457582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.457844 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.457821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwj6\" (UniqueName: \"kubernetes.io/projected/456b15ec-9505-47ae-97a2-ec32e7c811d0-kube-api-access-vzwj6\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzwj\" (UID: \"456b15ec-9505-47ae-97a2-ec32e7c811d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.570608 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.570576    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:27.927372 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.927347    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"]
Jul 16 21:56:27.929586 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:27.929558    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456b15ec_9505_47ae_97a2_ec32e7c811d0.slice/crio-d47987de2d12257d6fc71c33f740c0660ef90fc5833d39959cc913ca131ed3c8 WatchSource:0}: Error finding container d47987de2d12257d6fc71c33f740c0660ef90fc5833d39959cc913ca131ed3c8: Status 404 returned error can't find the container with id d47987de2d12257d6fc71c33f740c0660ef90fc5833d39959cc913ca131ed3c8
Jul 16 21:56:27.931601 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.931564    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:56:27.931671 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.931636    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:56:27.931671 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:27.931664    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:56:28.593885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:28.593851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" event={"ID":"456b15ec-9505-47ae-97a2-ec32e7c811d0","Type":"ContainerStarted","Data":"e5b3ce028a137bca70e1c95ab4360c16859a554930896ff3a6a65f0e3f405696"}
Jul 16 21:56:28.594266 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:28.593904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" event={"ID":"456b15ec-9505-47ae-97a2-ec32e7c811d0","Type":"ContainerStarted","Data":"d47987de2d12257d6fc71c33f740c0660ef90fc5833d39959cc913ca131ed3c8"}
Jul 16 21:56:28.625220 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:28.625168    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" podStartSLOduration=1.625150689 podStartE2EDuration="1.625150689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:28.622475589 +0000 UTC m=+547.132342663" watchObservedRunningTime="2026-07-16 21:56:28.625150689 +0000 UTC m=+547.135017770"
Jul 16 21:56:29.571002 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:29.570968    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:29.575886 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:29.575855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:29.599984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:29.599944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:29.600903 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:29.600874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:56:29.720427 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:29.720382    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:30.107140 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.107105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:30.107311 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.107153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:30.107311 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.107187    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.109879 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.109853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 21:56:30.172980 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.172944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6f7\" (UniqueName: \"kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.173149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.173029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.273446 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.273412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.273643 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.273521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6f7\" (UniqueName: \"kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.274101 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.274079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.284487 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.284437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6f7\" (UniqueName: \"kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7\") pod \"limitador-limitador-64c94fd64f-jgscx\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.417649 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.417563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:30.603818 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.603786    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:30.646247 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.646200    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:30.646247 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.646238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:30.650828 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.650795    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wjtr7\""
Jul 16 21:56:30.654229 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.654179    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:30.657932 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:30.657905    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b732a3_7e02_48e3_82fd_bbbb50ab0b2f.slice/crio-877e50925f4164ccc831023771ee0e10b6ec7bbbfd7ada9b864b2a4695ef9480 WatchSource:0}: Error finding container 877e50925f4164ccc831023771ee0e10b6ec7bbbfd7ada9b864b2a4695ef9480: Status 404 returned error can't find the container with id 877e50925f4164ccc831023771ee0e10b6ec7bbbfd7ada9b864b2a4695ef9480
Jul 16 21:56:30.678161 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.678063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqx75\" (UniqueName: \"kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75\") pod \"authorino-58cd955675-mdxxb\" (UID: \"0efe847b-f54e-43ef-a325-acc5ed27227c\") " pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:30.779784 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.779747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqx75\" (UniqueName: \"kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75\") pod \"authorino-58cd955675-mdxxb\" (UID: \"0efe847b-f54e-43ef-a325-acc5ed27227c\") " pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:30.791120 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.791093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqx75\" (UniqueName: \"kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75\") pod \"authorino-58cd955675-mdxxb\" (UID: \"0efe847b-f54e-43ef-a325-acc5ed27227c\") " pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:30.968703 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:30.968613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:31.136762 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:31.136730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:31.138073 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:31.138045    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0efe847b_f54e_43ef_a325_acc5ed27227c.slice/crio-3012bb32ced50247970242c8e3aacc22f1d5551172f54d5c0a5959b99c7c1c49 WatchSource:0}: Error finding container 3012bb32ced50247970242c8e3aacc22f1d5551172f54d5c0a5959b99c7c1c49: Status 404 returned error can't find the container with id 3012bb32ced50247970242c8e3aacc22f1d5551172f54d5c0a5959b99c7c1c49
Jul 16 21:56:31.618247 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:31.618109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" event={"ID":"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f","Type":"ContainerStarted","Data":"877e50925f4164ccc831023771ee0e10b6ec7bbbfd7ada9b864b2a4695ef9480"}
Jul 16 21:56:31.620979 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:31.620921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mdxxb" event={"ID":"0efe847b-f54e-43ef-a325-acc5ed27227c","Type":"ContainerStarted","Data":"3012bb32ced50247970242c8e3aacc22f1d5551172f54d5c0a5959b99c7c1c49"}
Jul 16 21:56:35.464613 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.464572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:35.648840 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.648806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" event={"ID":"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f","Type":"ContainerStarted","Data":"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"}
Jul 16 21:56:35.649077 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.648914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:35.651020 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.650979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mdxxb" event={"ID":"0efe847b-f54e-43ef-a325-acc5ed27227c","Type":"ContainerStarted","Data":"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"}
Jul 16 21:56:35.651177 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.651062    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-mdxxb" podUID="0efe847b-f54e-43ef-a325-acc5ed27227c" containerName="authorino" containerID="cri-o://a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65" gracePeriod=30
Jul 16 21:56:35.667508 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.667424    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" podStartSLOduration=1.812967912 podStartE2EDuration="6.667405508s" totalImagesPullingTime="4.854437596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:35.666082543 +0000 UTC m=+554.175949627" watchObservedRunningTime="2026-07-16 21:56:35.667405508 +0000 UTC m=+554.177272590"
Jul 16 21:56:35.682616 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.682561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-mdxxb" podStartSLOduration=1.3051641520000001 podStartE2EDuration="5.682544205s" totalImagesPullingTime="4.377380053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:35.681592645 +0000 UTC m=+554.191459743" watchObservedRunningTime="2026-07-16 21:56:35.682544205 +0000 UTC m=+554.192411287"
Jul 16 21:56:35.919365 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:35.919337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:36.033167 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.033082    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqx75\" (UniqueName: \"kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75\") pod \"0efe847b-f54e-43ef-a325-acc5ed27227c\" (UID: \"0efe847b-f54e-43ef-a325-acc5ed27227c\") "
Jul 16 21:56:36.035207 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.035184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75" (OuterVolumeSpecName: "kube-api-access-nqx75") pod "0efe847b-f54e-43ef-a325-acc5ed27227c" (UID: "0efe847b-f54e-43ef-a325-acc5ed27227c"). InnerVolumeSpecName "kube-api-access-nqx75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:36.134571 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.134533    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqx75\" (UniqueName: \"kubernetes.io/projected/0efe847b-f54e-43ef-a325-acc5ed27227c-kube-api-access-nqx75\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:36.657105 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.657067    2566 generic.go:356] "Generic (PLEG): container finished" podID="0efe847b-f54e-43ef-a325-acc5ed27227c" containerID="a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65" exitCode=0
Jul 16 21:56:36.657588 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.657124    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mdxxb"
Jul 16 21:56:36.657588 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.657149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mdxxb" event={"ID":"0efe847b-f54e-43ef-a325-acc5ed27227c","Type":"ContainerDied","Data":"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"}
Jul 16 21:56:36.657588 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.657191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mdxxb" event={"ID":"0efe847b-f54e-43ef-a325-acc5ed27227c","Type":"ContainerDied","Data":"3012bb32ced50247970242c8e3aacc22f1d5551172f54d5c0a5959b99c7c1c49"}
Jul 16 21:56:36.657588 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.657209    2566 scope.go:117] "RemoveContainer" containerID="a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"
Jul 16 21:56:36.667278 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.667260    2566 scope.go:117] "RemoveContainer" containerID="a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"
Jul 16 21:56:36.667610 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:56:36.667584    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65\": container with ID starting with a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65 not found: ID does not exist" containerID="a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"
Jul 16 21:56:36.667683 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.667619    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65"} err="failed to get container status \"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65\": rpc error: code = NotFound desc = could not find container \"a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65\": container with ID starting with a30e07d6eb0bb2c77c4440f6ce88dfd42ce3335a31abb0e6b01b41f313453a65 not found: ID does not exist"
Jul 16 21:56:36.681072 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.681039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:36.688162 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:36.688137    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-mdxxb"]
Jul 16 21:56:38.010663 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:38.010629    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efe847b-f54e-43ef-a325-acc5ed27227c" path="/var/lib/kubelet/pods/0efe847b-f54e-43ef-a325-acc5ed27227c/volumes"
Jul 16 21:56:44.463715 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:44.463684    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:44.464185 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:44.463896    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" podUID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" containerName="limitador" containerID="cri-o://24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff" gracePeriod=30
Jul 16 21:56:44.464617 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:44.464596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:45.412408 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.412387    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:45.523442 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.523399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file\") pod \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") "
Jul 16 21:56:45.523894 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.523552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm6f7\" (UniqueName: \"kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7\") pod \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\" (UID: \"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f\") "
Jul 16 21:56:45.523894 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.523752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file" (OuterVolumeSpecName: "config-file") pod "c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" (UID: "c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:45.523894 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.523821    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-config-file\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:45.525610 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.525554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7" (OuterVolumeSpecName: "kube-api-access-hm6f7") pod "c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" (UID: "c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f"). InnerVolumeSpecName "kube-api-access-hm6f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:45.625056 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.625016    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6f7\" (UniqueName: \"kubernetes.io/projected/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f-kube-api-access-hm6f7\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:56:45.703348 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.703309    2566 generic.go:356] "Generic (PLEG): container finished" podID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" containerID="24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff" exitCode=0
Jul 16 21:56:45.703556 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.703355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx"
Jul 16 21:56:45.703556 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.703396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" event={"ID":"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f","Type":"ContainerDied","Data":"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"}
Jul 16 21:56:45.703556 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.703448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jgscx" event={"ID":"c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f","Type":"ContainerDied","Data":"877e50925f4164ccc831023771ee0e10b6ec7bbbfd7ada9b864b2a4695ef9480"}
Jul 16 21:56:45.703556 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.703483    2566 scope.go:117] "RemoveContainer" containerID="24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"
Jul 16 21:56:45.715238 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.715217    2566 scope.go:117] "RemoveContainer" containerID="24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"
Jul 16 21:56:45.715572 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:56:45.715548    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff\": container with ID starting with 24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff not found: ID does not exist" containerID="24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"
Jul 16 21:56:45.715646 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.715580    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff"} err="failed to get container status \"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff\": rpc error: code = NotFound desc = could not find container \"24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff\": container with ID starting with 24202582607b5c1dab7281414c19208bab273e5a8704b155245203c68fd8a1ff not found: ID does not exist"
Jul 16 21:56:45.739551 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.739525    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:45.745156 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:45.745133    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jgscx"]
Jul 16 21:56:46.014611 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.012692    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" path="/var/lib/kubelet/pods/c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f/volumes"
Jul 16 21:56:46.431312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431233    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"]
Jul 16 21:56:46.431687 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431673    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" containerName="limitador"
Jul 16 21:56:46.431735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431689    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" containerName="limitador"
Jul 16 21:56:46.431735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431700    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efe847b-f54e-43ef-a325-acc5ed27227c" containerName="authorino"
Jul 16 21:56:46.431735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431706    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efe847b-f54e-43ef-a325-acc5ed27227c" containerName="authorino"
Jul 16 21:56:46.431830 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431781    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b732a3-7e02-48e3-82fd-bbbb50ab0b2f" containerName="limitador"
Jul 16 21:56:46.431830 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.431790    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efe847b-f54e-43ef-a325-acc5ed27227c" containerName="authorino"
Jul 16 21:56:46.435084 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.435053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.444625 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.444593    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 21:56:46.444767 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.444709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 21:56:46.445616 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.445597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 21:56:46.445764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.445601    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jf5pp\""
Jul 16 21:56:46.455585 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.455545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"]
Jul 16 21:56:46.535405 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.535361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-data\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.535405 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.535409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpvs\" (UniqueName: \"kubernetes.io/projected/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-kube-api-access-knpvs\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.637058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.637019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-data\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.637058 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.637063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpvs\" (UniqueName: \"kubernetes.io/projected/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-kube-api-access-knpvs\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.637507 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.637449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-data\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.653634 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.653609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpvs\" (UniqueName: \"kubernetes.io/projected/fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25-kube-api-access-knpvs\") pod \"postgres-69fb4594cf-cpgdw\" (UID: \"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.746129 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.746097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:46.890602 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:46.890574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"]
Jul 16 21:56:46.892431 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:46.892402    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc94e78e_1fee_4eb4_a2d1_a0709a7d5c25.slice/crio-682611a2bac2695912d632a5920eb17d4da80c22387722ed1701d2829bb9dc77 WatchSource:0}: Error finding container 682611a2bac2695912d632a5920eb17d4da80c22387722ed1701d2829bb9dc77: Status 404 returned error can't find the container with id 682611a2bac2695912d632a5920eb17d4da80c22387722ed1701d2829bb9dc77
Jul 16 21:56:47.720572 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:47.720485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw" event={"ID":"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25","Type":"ContainerStarted","Data":"682611a2bac2695912d632a5920eb17d4da80c22387722ed1701d2829bb9dc77"}
Jul 16 21:56:51.746499 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:51.746437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw" event={"ID":"fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25","Type":"ContainerStarted","Data":"0e2266bf451f03e380d28f43d0eaeeda549a4b0607343cabf1ef1bfd41ffc445"}
Jul 16 21:56:51.746933 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:51.746523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:51.770149 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:51.770097    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw" podStartSLOduration=1.111809273 podStartE2EDuration="5.770081679s" totalImagesPullingTime="4.658272406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:51.770023526 +0000 UTC m=+570.279890609" watchObservedRunningTime="2026-07-16 21:56:51.770081679 +0000 UTC m=+570.279948760"
Jul 16 21:56:57.780848 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:57.780816    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cpgdw"
Jul 16 21:56:58.330083 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.330048    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:56:58.336204 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.336182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:56:58.339989 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.339969    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wjtr7\""
Jul 16 21:56:58.350438 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.350417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:56:58.445120 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.445091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4998v\" (UniqueName: \"kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v\") pod \"authorino-5d4b67c446-svhsv\" (UID: \"ceb18667-208f-4074-8749-05f78f061c7b\") " pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:56:58.546445 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.546415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4998v\" (UniqueName: \"kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v\") pod \"authorino-5d4b67c446-svhsv\" (UID: \"ceb18667-208f-4074-8749-05f78f061c7b\") " pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:56:58.560023 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.559992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4998v\" (UniqueName: \"kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v\") pod \"authorino-5d4b67c446-svhsv\" (UID: \"ceb18667-208f-4074-8749-05f78f061c7b\") " pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:56:58.624691 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.624621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:56:58.624887 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.624875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:56:58.678248 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.678223    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74dfcd7f48-nfw8j"]
Jul 16 21:56:58.683534 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.683509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:58.696764 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.696702    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74dfcd7f48-nfw8j"]
Jul 16 21:56:58.775817 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.775792    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:56:58.778603 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:56:58.778577    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceb18667_208f_4074_8749_05f78f061c7b.slice/crio-716278a3b234f2cc2ecd9ec49d546d97e26143a8b0eeae2505c4f32618513ff4 WatchSource:0}: Error finding container 716278a3b234f2cc2ecd9ec49d546d97e26143a8b0eeae2505c4f32618513ff4: Status 404 returned error can't find the container with id 716278a3b234f2cc2ecd9ec49d546d97e26143a8b0eeae2505c4f32618513ff4
Jul 16 21:56:58.795527 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.795501    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74dfcd7f48-nfw8j"]
Jul 16 21:56:58.795873 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:56:58.795837    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4cf5k], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-74dfcd7f48-nfw8j" podUID="1b1fe0b4-9fe5-4aeb-84f2-36742283938d"
Jul 16 21:56:58.849550 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.849524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cf5k\" (UniqueName: \"kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k\") pod \"authorino-74dfcd7f48-nfw8j\" (UID: \"1b1fe0b4-9fe5-4aeb-84f2-36742283938d\") " pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:58.950382 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.950295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cf5k\" (UniqueName: \"kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k\") pod \"authorino-74dfcd7f48-nfw8j\" (UID: \"1b1fe0b4-9fe5-4aeb-84f2-36742283938d\") " pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:58.959483 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:58.959442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cf5k\" (UniqueName: \"kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k\") pod \"authorino-74dfcd7f48-nfw8j\" (UID: \"1b1fe0b4-9fe5-4aeb-84f2-36742283938d\") " pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:59.789003 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.788975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:59.789181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.789019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-svhsv" event={"ID":"ceb18667-208f-4074-8749-05f78f061c7b","Type":"ContainerStarted","Data":"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"}
Jul 16 21:56:59.789181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.789038    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-svhsv" podUID="ceb18667-208f-4074-8749-05f78f061c7b" containerName="authorino" containerID="cri-o://7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1" gracePeriod=30
Jul 16 21:56:59.789181 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.789066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-svhsv" event={"ID":"ceb18667-208f-4074-8749-05f78f061c7b","Type":"ContainerStarted","Data":"716278a3b234f2cc2ecd9ec49d546d97e26143a8b0eeae2505c4f32618513ff4"}
Jul 16 21:56:59.829728 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.829706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:56:59.830067 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.830027    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-svhsv" podStartSLOduration=1.467060085 podStartE2EDuration="1.830013437s" totalImagesPullingTime="362.953352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:59.828749296 +0000 UTC m=+578.338616379" watchObservedRunningTime="2026-07-16 21:56:59.830013437 +0000 UTC m=+578.339880519"
Jul 16 21:56:59.958933 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.958893    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cf5k\" (UniqueName: \"kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k\") pod \"1b1fe0b4-9fe5-4aeb-84f2-36742283938d\" (UID: \"1b1fe0b4-9fe5-4aeb-84f2-36742283938d\") "
Jul 16 21:56:59.961066 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:56:59.961033    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k" (OuterVolumeSpecName: "kube-api-access-4cf5k") pod "1b1fe0b4-9fe5-4aeb-84f2-36742283938d" (UID: "1b1fe0b4-9fe5-4aeb-84f2-36742283938d"). InnerVolumeSpecName "kube-api-access-4cf5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:00.059553 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.059523    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cf5k\" (UniqueName: \"kubernetes.io/projected/1b1fe0b4-9fe5-4aeb-84f2-36742283938d-kube-api-access-4cf5k\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:57:00.071272 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.071244    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:57:00.160007 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.159975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4998v\" (UniqueName: \"kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v\") pod \"ceb18667-208f-4074-8749-05f78f061c7b\" (UID: \"ceb18667-208f-4074-8749-05f78f061c7b\") "
Jul 16 21:57:00.161922 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.161895    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v" (OuterVolumeSpecName: "kube-api-access-4998v") pod "ceb18667-208f-4074-8749-05f78f061c7b" (UID: "ceb18667-208f-4074-8749-05f78f061c7b"). InnerVolumeSpecName "kube-api-access-4998v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:00.260765 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.260735    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4998v\" (UniqueName: \"kubernetes.io/projected/ceb18667-208f-4074-8749-05f78f061c7b-kube-api-access-4998v\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:57:00.795734 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795688    2566 generic.go:356] "Generic (PLEG): container finished" podID="ceb18667-208f-4074-8749-05f78f061c7b" containerID="7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1" exitCode=0
Jul 16 21:57:00.795920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-svhsv"
Jul 16 21:57:00.795920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-svhsv" event={"ID":"ceb18667-208f-4074-8749-05f78f061c7b","Type":"ContainerDied","Data":"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"}
Jul 16 21:57:00.795920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-svhsv" event={"ID":"ceb18667-208f-4074-8749-05f78f061c7b","Type":"ContainerDied","Data":"716278a3b234f2cc2ecd9ec49d546d97e26143a8b0eeae2505c4f32618513ff4"}
Jul 16 21:57:00.795920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795853    2566 scope.go:117] "RemoveContainer" containerID="7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"
Jul 16 21:57:00.796122 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.795977    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dfcd7f48-nfw8j"
Jul 16 21:57:00.805581 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.805562    2566 scope.go:117] "RemoveContainer" containerID="7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"
Jul 16 21:57:00.805800 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:57:00.805782    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1\": container with ID starting with 7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1 not found: ID does not exist" containerID="7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"
Jul 16 21:57:00.805865 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.805807    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1"} err="failed to get container status \"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1\": rpc error: code = NotFound desc = could not find container \"7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1\": container with ID starting with 7c2965b6e0c11383b268f83265fd6ad47c0ee536118cda6fa16df8f74aa4c6c1 not found: ID does not exist"
Jul 16 21:57:00.833285 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.833226    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74dfcd7f48-nfw8j"]
Jul 16 21:57:00.841514 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.841491    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74dfcd7f48-nfw8j"]
Jul 16 21:57:00.858614 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.858586    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:57:00.871367 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:00.871341    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-svhsv"]
Jul 16 21:57:02.012139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:02.012104    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1fe0b4-9fe5-4aeb-84f2-36742283938d" path="/var/lib/kubelet/pods/1b1fe0b4-9fe5-4aeb-84f2-36742283938d/volumes"
Jul 16 21:57:02.012599 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:02.012327    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb18667-208f-4074-8749-05f78f061c7b" path="/var/lib/kubelet/pods/ceb18667-208f-4074-8749-05f78f061c7b/volumes"
Jul 16 21:57:05.421005 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.420968    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:05.421508 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.421487    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb18667-208f-4074-8749-05f78f061c7b" containerName="authorino"
Jul 16 21:57:05.421668 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.421648    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb18667-208f-4074-8749-05f78f061c7b" containerName="authorino"
Jul 16 21:57:05.421844 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.421828    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb18667-208f-4074-8749-05f78f061c7b" containerName="authorino"
Jul 16 21:57:05.441013 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.440983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:05.441185 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.441108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.443858 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.443827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tp54f\""
Jul 16 21:57:05.443858 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.443853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 21:57:05.444144 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.444121    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 21:57:05.507687 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.507651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.507912 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.507759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx428\" (UniqueName: \"kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.592812 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.592776    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 21:57:05.595533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.595512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.608491 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.608438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.608619 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.608532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx428\" (UniqueName: \"kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.608619 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.608582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk7f\" (UniqueName: \"kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.608733 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.608628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.608836 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:57:05.608818    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 16 21:57:05.608909 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:57:05.608885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert podName:1c32923f-5e90-46f9-a71d-4fc145791c6b nodeName:}" failed. No retries permitted until 2026-07-16 21:57:06.108863864 +0000 UTC m=+584.618730935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert") pod "maas-controller-6896459b7d-4bkbb" (UID: "1c32923f-5e90-46f9-a71d-4fc145791c6b") : secret "maas-controller-webhook-cert" not found
Jul 16 21:57:05.610787 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.610763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 21:57:05.623525 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.623499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx428\" (UniqueName: \"kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:05.709227 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.709161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.709227 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.709212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk7f\" (UniqueName: \"kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.711538 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.711510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.717816 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.717795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk7f\" (UniqueName: \"kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f\") pod \"maas-controller-58f7bcf9f5-dp9qx\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") " pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:05.905904 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:05.905872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:06.041394 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.041362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 21:57:06.042164 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:57:06.042135    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c83f31f_29a3_4f88_953b_b37eb78a3940.slice/crio-6610c2404833eb9083a8a64f84f2f6609e47d7ea957348c02e39722fc9f156b3 WatchSource:0}: Error finding container 6610c2404833eb9083a8a64f84f2f6609e47d7ea957348c02e39722fc9f156b3: Status 404 returned error can't find the container with id 6610c2404833eb9083a8a64f84f2f6609e47d7ea957348c02e39722fc9f156b3
Jul 16 21:57:06.112066 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.112037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:06.114384 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.114361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") pod \"maas-controller-6896459b7d-4bkbb\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") " pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:06.351833 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.351794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:06.501736 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.501713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:06.504056 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:57:06.504029    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c32923f_5e90_46f9_a71d_4fc145791c6b.slice/crio-6b9fa5f976b9d1d1313ab1689e9cee2661c41cef4923b7f618552f87c3e0f580 WatchSource:0}: Error finding container 6b9fa5f976b9d1d1313ab1689e9cee2661c41cef4923b7f618552f87c3e0f580: Status 404 returned error can't find the container with id 6b9fa5f976b9d1d1313ab1689e9cee2661c41cef4923b7f618552f87c3e0f580
Jul 16 21:57:06.833094 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.833018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6896459b7d-4bkbb" event={"ID":"1c32923f-5e90-46f9-a71d-4fc145791c6b","Type":"ContainerStarted","Data":"6b9fa5f976b9d1d1313ab1689e9cee2661c41cef4923b7f618552f87c3e0f580"}
Jul 16 21:57:06.834642 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:06.834603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" event={"ID":"4c83f31f-29a3-4f88-953b-b37eb78a3940","Type":"ContainerStarted","Data":"6610c2404833eb9083a8a64f84f2f6609e47d7ea957348c02e39722fc9f156b3"}
Jul 16 21:57:09.855135 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.855087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" event={"ID":"4c83f31f-29a3-4f88-953b-b37eb78a3940","Type":"ContainerStarted","Data":"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"}
Jul 16 21:57:09.855699 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.855509    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:09.856862 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.856831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6896459b7d-4bkbb" event={"ID":"1c32923f-5e90-46f9-a71d-4fc145791c6b","Type":"ContainerStarted","Data":"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"}
Jul 16 21:57:09.856994 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.856955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:09.874191 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.874142    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" podStartSLOduration=2.07822837 podStartE2EDuration="4.874126936s" totalImagesPullingTime="2.795898566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:09.873518331 +0000 UTC m=+588.383385414" watchObservedRunningTime="2026-07-16 21:57:09.874126936 +0000 UTC m=+588.383994018"
Jul 16 21:57:09.892564 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:09.892510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6896459b7d-4bkbb" podStartSLOduration=2.548658969 podStartE2EDuration="4.892491874s" totalImagesPullingTime="2.343832905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:09.891361898 +0000 UTC m=+588.401228981" watchObservedRunningTime="2026-07-16 21:57:09.892491874 +0000 UTC m=+588.402358949"
Jul 16 21:57:20.863826 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:20.863783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:20.864288 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:20.863864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 21:57:20.932946 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:20.932911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:20.933161 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:20.933136    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6896459b7d-4bkbb" podUID="1c32923f-5e90-46f9-a71d-4fc145791c6b" containerName="manager" containerID="cri-o://bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd" gracePeriod=30
Jul 16 21:57:21.187134 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.187109    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:21.252889 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.252857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx428\" (UniqueName: \"kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428\") pod \"1c32923f-5e90-46f9-a71d-4fc145791c6b\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") "
Jul 16 21:57:21.253043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.252923    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") pod \"1c32923f-5e90-46f9-a71d-4fc145791c6b\" (UID: \"1c32923f-5e90-46f9-a71d-4fc145791c6b\") "
Jul 16 21:57:21.255102 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.255068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1c32923f-5e90-46f9-a71d-4fc145791c6b" (UID: "1c32923f-5e90-46f9-a71d-4fc145791c6b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:57:21.255218 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.255120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428" (OuterVolumeSpecName: "kube-api-access-jx428") pod "1c32923f-5e90-46f9-a71d-4fc145791c6b" (UID: "1c32923f-5e90-46f9-a71d-4fc145791c6b"). InnerVolumeSpecName "kube-api-access-jx428". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:21.354251 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.354224    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c32923f-5e90-46f9-a71d-4fc145791c6b-webhook-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:57:21.354251 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.354249    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx428\" (UniqueName: \"kubernetes.io/projected/1c32923f-5e90-46f9-a71d-4fc145791c6b-kube-api-access-jx428\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 21:57:21.921274 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.921245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:57:21.923148 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.923118    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 21:57:21.927221 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.927187    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c32923f-5e90-46f9-a71d-4fc145791c6b" containerID="bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd" exitCode=0
Jul 16 21:57:21.927385 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.927217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6896459b7d-4bkbb" event={"ID":"1c32923f-5e90-46f9-a71d-4fc145791c6b","Type":"ContainerDied","Data":"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"}
Jul 16 21:57:21.927385 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.927238    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6896459b7d-4bkbb"
Jul 16 21:57:21.927385 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.927284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6896459b7d-4bkbb" event={"ID":"1c32923f-5e90-46f9-a71d-4fc145791c6b","Type":"ContainerDied","Data":"6b9fa5f976b9d1d1313ab1689e9cee2661c41cef4923b7f618552f87c3e0f580"}
Jul 16 21:57:21.927385 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.927315    2566 scope.go:117] "RemoveContainer" containerID="bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"
Jul 16 21:57:21.937333 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.937129    2566 scope.go:117] "RemoveContainer" containerID="bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"
Jul 16 21:57:21.937432 ip-10-0-140-81 kubenswrapper[2566]: E0716 21:57:21.937413    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd\": container with ID starting with bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd not found: ID does not exist" containerID="bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"
Jul 16 21:57:21.937510 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.937442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd"} err="failed to get container status \"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd\": rpc error: code = NotFound desc = could not find container \"bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd\": container with ID starting with bd64f5c98adaac17eb63fc7e37f490e9fc2554e0569c33de0c4c2635e15755fd not found: ID does not exist"
Jul 16 21:57:21.949753 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.949718    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:21.955908 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:21.955866    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6896459b7d-4bkbb"]
Jul 16 21:57:22.011624 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:22.011583    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c32923f-5e90-46f9-a71d-4fc145791c6b" path="/var/lib/kubelet/pods/1c32923f-5e90-46f9-a71d-4fc145791c6b/volumes"
Jul 16 21:57:41.491531 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.491493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"]
Jul 16 21:57:41.492027 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.491957    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c32923f-5e90-46f9-a71d-4fc145791c6b" containerName="manager"
Jul 16 21:57:41.492027 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.491970    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c32923f-5e90-46f9-a71d-4fc145791c6b" containerName="manager"
Jul 16 21:57:41.492104 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.492049    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c32923f-5e90-46f9-a71d-4fc145791c6b" containerName="manager"
Jul 16 21:57:41.497560 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.497533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.500804 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.500781    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 16 21:57:41.515566 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.515534    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"]
Jul 16 21:57:41.632950 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.632915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5510fa8c-ea98-4a48-87fb-2f00c661465f-openshift-service-ca\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.633139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.632959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftqr\" (UniqueName: \"kubernetes.io/projected/5510fa8c-ea98-4a48-87fb-2f00c661465f-kube-api-access-mftqr\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.633139 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.633004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5510fa8c-ea98-4a48-87fb-2f00c661465f-maas-api-tls\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.734072 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.734043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5510fa8c-ea98-4a48-87fb-2f00c661465f-openshift-service-ca\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.734246 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.734080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftqr\" (UniqueName: \"kubernetes.io/projected/5510fa8c-ea98-4a48-87fb-2f00c661465f-kube-api-access-mftqr\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.734246 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.734221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5510fa8c-ea98-4a48-87fb-2f00c661465f-maas-api-tls\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.734844 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.734817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5510fa8c-ea98-4a48-87fb-2f00c661465f-openshift-service-ca\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.736735 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.736713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5510fa8c-ea98-4a48-87fb-2f00c661465f-maas-api-tls\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.749892 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.749830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftqr\" (UniqueName: \"kubernetes.io/projected/5510fa8c-ea98-4a48-87fb-2f00c661465f-kube-api-access-mftqr\") pod \"maas-api-77c4c984f4-pqbgm\" (UID: \"5510fa8c-ea98-4a48-87fb-2f00c661465f\") " pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.807726 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.807690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:41.969188 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:41.969158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"]
Jul 16 21:57:41.971485 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:57:41.971438    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5510fa8c_ea98_4a48_87fb_2f00c661465f.slice/crio-28e2c3a5f9bacd452c5460d904ab973cd74350ed685bc52d7c4f7614bcbd5d20 WatchSource:0}: Error finding container 28e2c3a5f9bacd452c5460d904ab973cd74350ed685bc52d7c4f7614bcbd5d20: Status 404 returned error can't find the container with id 28e2c3a5f9bacd452c5460d904ab973cd74350ed685bc52d7c4f7614bcbd5d20
Jul 16 21:57:42.027744 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.027667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm" event={"ID":"5510fa8c-ea98-4a48-87fb-2f00c661465f","Type":"ContainerStarted","Data":"28e2c3a5f9bacd452c5460d904ab973cd74350ed685bc52d7c4f7614bcbd5d20"}
Jul 16 21:57:42.541965 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.541932    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"]
Jul 16 21:57:42.546369 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.546344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.553732 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.553713    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 21:57:42.554203 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.554185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4dkhq\""
Jul 16 21:57:42.563487 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.563442    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6s658"]
Jul 16 21:57:42.566267 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.566253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.571791 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.571768    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"]
Jul 16 21:57:42.601384 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.601337    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6s658"]
Jul 16 21:57:42.642533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.642244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9gm\" (UniqueName: \"kubernetes.io/projected/b65f15b0-318e-41ba-8fbd-6bb7d008343d-kube-api-access-2z9gm\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.642533 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.642338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65f15b0-318e-41ba-8fbd-6bb7d008343d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.743326 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.743294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9gm\" (UniqueName: \"kubernetes.io/projected/b65f15b0-318e-41ba-8fbd-6bb7d008343d-kube-api-access-2z9gm\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.743548 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.743349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65f15b0-318e-41ba-8fbd-6bb7d008343d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.743548 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.743400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.743548 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.743442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cx84\" (UniqueName: \"kubernetes.io/projected/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-kube-api-access-4cx84\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.744210 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.744184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b65f15b0-318e-41ba-8fbd-6bb7d008343d-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.754118 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.754088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9gm\" (UniqueName: \"kubernetes.io/projected/b65f15b0-318e-41ba-8fbd-6bb7d008343d-kube-api-access-2z9gm\") pod \"payload-pre-processing-7677b774bb-6jvh5\" (UID: \"b65f15b0-318e-41ba-8fbd-6bb7d008343d\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.844638 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.844546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.844638 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.844607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cx84\" (UniqueName: \"kubernetes.io/projected/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-kube-api-access-4cx84\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.845400 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.845375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.857570 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.857544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:42.867773 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.867754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cx84\" (UniqueName: \"kubernetes.io/projected/5c2dae41-16b6-463c-a7b1-8ce8579a0c60-kube-api-access-4cx84\") pod \"payload-processing-5bf47cd988-6s658\" (UID: \"5c2dae41-16b6-463c-a7b1-8ce8579a0c60\") " pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:42.876550 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:42.876525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:43.039979 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:43.039740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"]
Jul 16 21:57:43.043191 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:57:43.043161    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65f15b0_318e_41ba_8fbd_6bb7d008343d.slice/crio-1f5691b8ce760af04e41b64ab91ef3aa58fb7202596bfe20f09388f12ac8fabb WatchSource:0}: Error finding container 1f5691b8ce760af04e41b64ab91ef3aa58fb7202596bfe20f09388f12ac8fabb: Status 404 returned error can't find the container with id 1f5691b8ce760af04e41b64ab91ef3aa58fb7202596bfe20f09388f12ac8fabb
Jul 16 21:57:43.073273 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:57:43.073246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2dae41_16b6_463c_a7b1_8ce8579a0c60.slice/crio-653094465e18878ac86276dd1f8c2817ad65265b8822710486f6ae83806c7b35 WatchSource:0}: Error finding container 653094465e18878ac86276dd1f8c2817ad65265b8822710486f6ae83806c7b35: Status 404 returned error can't find the container with id 653094465e18878ac86276dd1f8c2817ad65265b8822710486f6ae83806c7b35
Jul 16 21:57:43.078148 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:43.078128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6s658"]
Jul 16 21:57:44.059393 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:44.059257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5" event={"ID":"b65f15b0-318e-41ba-8fbd-6bb7d008343d","Type":"ContainerStarted","Data":"1f5691b8ce760af04e41b64ab91ef3aa58fb7202596bfe20f09388f12ac8fabb"}
Jul 16 21:57:44.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:44.061475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm" event={"ID":"5510fa8c-ea98-4a48-87fb-2f00c661465f","Type":"ContainerStarted","Data":"65052c34e301f1ba41ef5c113841acdbb7c120817cf5689e09cf502b27370764"}
Jul 16 21:57:44.061748 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:44.061560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:44.066287 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:44.066219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-6s658" event={"ID":"5c2dae41-16b6-463c-a7b1-8ce8579a0c60","Type":"ContainerStarted","Data":"653094465e18878ac86276dd1f8c2817ad65265b8822710486f6ae83806c7b35"}
Jul 16 21:57:44.085984 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:44.085928    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm" podStartSLOduration=1.302504596 podStartE2EDuration="3.085902315s" totalImagesPullingTime="1.783397719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:44.085322987 +0000 UTC m=+622.595190070" watchObservedRunningTime="2026-07-16 21:57:44.085902315 +0000 UTC m=+622.595769398"
Jul 16 21:57:46.078179 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:46.078137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5" event={"ID":"b65f15b0-318e-41ba-8fbd-6bb7d008343d","Type":"ContainerStarted","Data":"cf30f56f957a8af5f411b94c337d9ce7ab2832667e3ca82a8b5748d671ccceae"}
Jul 16 21:57:46.078695 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:46.078308    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:46.079756 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:46.079730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-6s658" event={"ID":"5c2dae41-16b6-463c-a7b1-8ce8579a0c60","Type":"ContainerStarted","Data":"0f4686414de74a2b14e863e85a38124eb5b914110fd23799e58c91811efa4dd4"}
Jul 16 21:57:46.079885 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:46.079855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:57:46.104543 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:46.104501    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5" podStartSLOduration=1.605151021 podStartE2EDuration="4.104487268s" totalImagesPullingTime="2.499336247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:46.102001733 +0000 UTC m=+624.611868814" watchObservedRunningTime="2026-07-16 21:57:46.104487268 +0000 UTC m=+624.614354351"
Jul 16 21:57:50.077491 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:50.077438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77c4c984f4-pqbgm"
Jul 16 21:57:50.095910 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:50.095859    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-6s658" podStartSLOduration=5.652002161 podStartE2EDuration="8.095843264s" totalImagesPullingTime="2.443841103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:46.137485968 +0000 UTC m=+624.647353048" watchObservedRunningTime="2026-07-16 21:57:50.095843264 +0000 UTC m=+628.605710348"
Jul 16 21:57:51.109466 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.109421    2566 generic.go:356] "Generic (PLEG): container finished" podID="456b15ec-9505-47ae-97a2-ec32e7c811d0" containerID="e5b3ce028a137bca70e1c95ab4360c16859a554930896ff3a6a65f0e3f405696" exitCode=0
Jul 16 21:57:51.109886 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.109493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" event={"ID":"456b15ec-9505-47ae-97a2-ec32e7c811d0","Type":"ContainerDied","Data":"e5b3ce028a137bca70e1c95ab4360c16859a554930896ff3a6a65f0e3f405696"}
Jul 16 21:57:51.109886 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.109854    2566 scope.go:117] "RemoveContainer" containerID="e5b3ce028a137bca70e1c95ab4360c16859a554930896ff3a6a65f0e3f405696"
Jul 16 21:57:51.110891 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.110854    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:57:51.110967 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.110948    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:57:51.111021 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:51.110998    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 21:57:52.116959 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:52.116921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" event={"ID":"456b15ec-9505-47ae-97a2-ec32e7c811d0","Type":"ContainerStarted","Data":"d2cc0eb5b0c816a3a0dff20b35e7286550a69dcdd929190815e6e9eae700e856"}
Jul 16 21:57:52.571309 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:52.571276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:57:52.571595 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:52.571568    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 16 21:57:52.571668 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:52.571621    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" podUID="456b15ec-9505-47ae-97a2-ec32e7c811d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 16 21:57:53.571779 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:53.571738    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 16 21:57:53.572230 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:53.571802    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" podUID="456b15ec-9505-47ae-97a2-ec32e7c811d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 16 21:57:54.571276 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:54.571204    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 16 21:57:54.571276 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:54.571266    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" podUID="456b15ec-9505-47ae-97a2-ec32e7c811d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 16 21:57:55.571254 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:55.571217    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Jul 16 21:57:55.571644 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:55.571269    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj" podUID="456b15ec-9505-47ae-97a2-ec32e7c811d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Jul 16 21:57:56.574936 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:56.574908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:57:56.575288 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:56.575267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:57:56.575927 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:56.575903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzwj"
Jul 16 21:57:57.086727 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:57.086694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-6jvh5"
Jul 16 21:57:57.086925 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:57:57.086820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-6s658"
Jul 16 21:58:02.669722 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.669682    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"]
Jul 16 21:58:02.674917 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.674894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.677474 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.677435    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 21:58:02.678494 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.678479    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 16 21:58:02.678582 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.678484    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 21:58:02.678582 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.678568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5gqbs\""
Jul 16 21:58:02.686127 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.686106    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"]
Jul 16 21:58:02.740660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.740660 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.740837 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.740837 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.740837 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7jz\" (UniqueName: \"kubernetes.io/projected/bacae03c-47ad-4ace-889f-5db2c21a623f-kube-api-access-mh7jz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.740931 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.740847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacae03c-47ad-4ace-889f-5db2c21a623f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.841995 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.841962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7jz\" (UniqueName: \"kubernetes.io/projected/bacae03c-47ad-4ace-889f-5db2c21a623f-kube-api-access-mh7jz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.841995 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacae03c-47ad-4ace-889f-5db2c21a623f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842233 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842658 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842747 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.842747 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.842676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.844371 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.844354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bacae03c-47ad-4ace-889f-5db2c21a623f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.844548 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.844531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacae03c-47ad-4ace-889f-5db2c21a623f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.850690 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.850671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7jz\" (UniqueName: \"kubernetes.io/projected/bacae03c-47ad-4ace-889f-5db2c21a623f-kube-api-access-mh7jz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh\" (UID: \"bacae03c-47ad-4ace-889f-5db2c21a623f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:02.986312 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:02.986245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:03.121932 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:03.121913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"]
Jul 16 21:58:03.124522 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:58:03.124488    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacae03c_47ad_4ace_889f_5db2c21a623f.slice/crio-ae26be8e0433879b4d3b3e6e13d55a79bb353f0703f09adb5c546ef59180d35e WatchSource:0}: Error finding container ae26be8e0433879b4d3b3e6e13d55a79bb353f0703f09adb5c546ef59180d35e: Status 404 returned error can't find the container with id ae26be8e0433879b4d3b3e6e13d55a79bb353f0703f09adb5c546ef59180d35e
Jul 16 21:58:03.126065 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:03.126044    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:58:03.183993 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:03.183959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh" event={"ID":"bacae03c-47ad-4ace-889f-5db2c21a623f","Type":"ContainerStarted","Data":"ae26be8e0433879b4d3b3e6e13d55a79bb353f0703f09adb5c546ef59180d35e"}
Jul 16 21:58:05.197341 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:05.197309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh" event={"ID":"bacae03c-47ad-4ace-889f-5db2c21a623f","Type":"ContainerStarted","Data":"84c455df1011ba5464ece171a3d3937977cc0447163ea811f8c73ef952126771"}
Jul 16 21:58:05.216317 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:05.216268    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh" podStartSLOduration=2.004969047 podStartE2EDuration="3.216252055s" totalImagesPullingTime="1.211283008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:05.214967816 +0000 UTC m=+643.724834897" watchObservedRunningTime="2026-07-16 21:58:05.216252055 +0000 UTC m=+643.726119136"
Jul 16 21:58:12.987386 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:12.987346    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:12.987983 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:12.987413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:13.000202 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:13.000169    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:13.259510 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:13.259476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh"
Jul 16 21:58:35.888741 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.888708    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"]
Jul 16 21:58:35.893664 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.893642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.896269 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.896244    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 16 21:58:35.902920 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.902895    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"]
Jul 16 21:58:35.956544 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.956684 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs86\" (UniqueName: \"kubernetes.io/projected/948df7eb-6796-4896-877e-c3f7c46ae6c1-kube-api-access-mhs86\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.956684 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948df7eb-6796-4896-877e-c3f7c46ae6c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.956684 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.956812 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:35.956812 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:35.956764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.057677 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948df7eb-6796-4896-877e-c3f7c46ae6c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.057850 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.057850 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.057850 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.057850 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.058056 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.057868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs86\" (UniqueName: \"kubernetes.io/projected/948df7eb-6796-4896-877e-c3f7c46ae6c1-kube-api-access-mhs86\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.058210 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.058178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.058327 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.058237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.058391 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.058325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.060043 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.060023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948df7eb-6796-4896-877e-c3f7c46ae6c1-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.060330 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.060311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948df7eb-6796-4896-877e-c3f7c46ae6c1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.066825 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.066801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs86\" (UniqueName: \"kubernetes.io/projected/948df7eb-6796-4896-877e-c3f7c46ae6c1-kube-api-access-mhs86\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9tddb\" (UID: \"948df7eb-6796-4896-877e-c3f7c46ae6c1\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.205237 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.205151    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:36.341997 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.341967    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"]
Jul 16 21:58:36.343488 ip-10-0-140-81 kubenswrapper[2566]: W0716 21:58:36.343419    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948df7eb_6796_4896_877e_c3f7c46ae6c1.slice/crio-ea1395608bf80d1d2d49895b194e0ae5a180162c1788f7375bb52df00e36bed9 WatchSource:0}: Error finding container ea1395608bf80d1d2d49895b194e0ae5a180162c1788f7375bb52df00e36bed9: Status 404 returned error can't find the container with id ea1395608bf80d1d2d49895b194e0ae5a180162c1788f7375bb52df00e36bed9
Jul 16 21:58:36.366104 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:36.366073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb" event={"ID":"948df7eb-6796-4896-877e-c3f7c46ae6c1","Type":"ContainerStarted","Data":"ea1395608bf80d1d2d49895b194e0ae5a180162c1788f7375bb52df00e36bed9"}
Jul 16 21:58:37.372815 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:37.372779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb" event={"ID":"948df7eb-6796-4896-877e-c3f7c46ae6c1","Type":"ContainerStarted","Data":"694851b103106e8dbd9aae1e3371fe8611989665b5dbeb6ba8a112d949f4b45f"}
Jul 16 21:58:37.391870 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:37.391823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb" podStartSLOduration=2.213792704 podStartE2EDuration="2.391807881s" totalImagesPullingTime="178.015177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:37.390733183 +0000 UTC m=+675.900600290" watchObservedRunningTime="2026-07-16 21:58:37.391807881 +0000 UTC m=+675.901674963"
Jul 16 21:58:46.205919 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:46.205881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:46.206410 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:46.205938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:46.218806 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:46.218780    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 21:58:46.431468 ip-10-0-140-81 kubenswrapper[2566]: I0716 21:58:46.431423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9tddb"
Jul 16 22:00:00.165122 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.165088    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"]
Jul 16 22:00:00.168665 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.168649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"
Jul 16 22:00:00.172691 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.172668    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:00:00.178843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.178822    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"
Jul 16 22:00:00.180156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.180127    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"]
Jul 16 22:00:00.532923 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.532897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"]
Jul 16 22:00:00.535027 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:00:00.534998    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode666691e_b48f_4e70_8123_7418cd485a74.slice/crio-6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e WatchSource:0}: Error finding container 6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e: Status 404 returned error can't find the container with id 6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e
Jul 16 22:00:00.810505 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:00.810399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q" event={"ID":"e666691e-b48f-4e70-8123-7418cd485a74","Type":"ContainerStarted","Data":"6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e"}
Jul 16 22:00:16.897367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:16.897265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q" event={"ID":"e666691e-b48f-4e70-8123-7418cd485a74","Type":"ContainerStarted","Data":"f257b016562bb3fab292c5be996e00716f139cfb1dd0b022472f53610cabd589"}
Jul 16 22:00:16.914052 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:16.913995    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q" podStartSLOduration=0.807200692 podStartE2EDuration="16.913977263s" totalImagesPullingTime="16.106776571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:16.912132839 +0000 UTC m=+775.421999921" watchObservedRunningTime="2026-07-16 22:00:16.913977263 +0000 UTC m=+775.423844389"
Jul 16 22:00:17.904106 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:17.904017    2566 generic.go:356] "Generic (PLEG): container finished" podID="e666691e-b48f-4e70-8123-7418cd485a74" containerID="f257b016562bb3fab292c5be996e00716f139cfb1dd0b022472f53610cabd589" exitCode=0
Jul 16 22:00:17.904482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:17.904108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q" event={"ID":"e666691e-b48f-4e70-8123-7418cd485a74","Type":"ContainerDied","Data":"f257b016562bb3fab292c5be996e00716f139cfb1dd0b022472f53610cabd589"}
Jul 16 22:00:19.044628 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:19.044600    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"
Jul 16 22:00:19.915676 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:19.915649    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"
Jul 16 22:00:19.915676 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:19.915669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q" event={"ID":"e666691e-b48f-4e70-8123-7418cd485a74","Type":"ContainerDied","Data":"6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e"}
Jul 16 22:00:19.915880 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:00:19.915745    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a1ade6da278d3452aecf836162368ecfb7ec81ad35487a9dad5e2eeff2ca95e"
Jul 16 22:01:03.692659 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:03.692624    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 22:01:03.693239 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:03.692886    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" podUID="4c83f31f-29a3-4f88-953b-b37eb78a3940" containerName="manager" containerID="cri-o://6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a" gracePeriod=30
Jul 16 22:01:03.955172 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:03.955114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 22:01:04.056133 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.056091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lk7f\" (UniqueName: \"kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f\") pod \"4c83f31f-29a3-4f88-953b-b37eb78a3940\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") "
Jul 16 22:01:04.056301 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.056238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert\") pod \"4c83f31f-29a3-4f88-953b-b37eb78a3940\" (UID: \"4c83f31f-29a3-4f88-953b-b37eb78a3940\") "
Jul 16 22:01:04.058137 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.058108    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f" (OuterVolumeSpecName: "kube-api-access-5lk7f") pod "4c83f31f-29a3-4f88-953b-b37eb78a3940" (UID: "4c83f31f-29a3-4f88-953b-b37eb78a3940"). InnerVolumeSpecName "kube-api-access-5lk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:01:04.058310 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.058286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4c83f31f-29a3-4f88-953b-b37eb78a3940" (UID: "4c83f31f-29a3-4f88-953b-b37eb78a3940"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:01:04.150414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.150378    2566 generic.go:356] "Generic (PLEG): container finished" podID="4c83f31f-29a3-4f88-953b-b37eb78a3940" containerID="6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a" exitCode=0
Jul 16 22:01:04.150596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.150435    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx"
Jul 16 22:01:04.150596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.150497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" event={"ID":"4c83f31f-29a3-4f88-953b-b37eb78a3940","Type":"ContainerDied","Data":"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"}
Jul 16 22:01:04.150596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.150552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-dp9qx" event={"ID":"4c83f31f-29a3-4f88-953b-b37eb78a3940","Type":"ContainerDied","Data":"6610c2404833eb9083a8a64f84f2f6609e47d7ea957348c02e39722fc9f156b3"}
Jul 16 22:01:04.150596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.150578    2566 scope.go:117] "RemoveContainer" containerID="6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"
Jul 16 22:01:04.157007 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.156985    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c83f31f-29a3-4f88-953b-b37eb78a3940-webhook-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:01:04.157007 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.157008    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk7f\" (UniqueName: \"kubernetes.io/projected/4c83f31f-29a3-4f88-953b-b37eb78a3940-kube-api-access-5lk7f\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:01:04.160146 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.160130    2566 scope.go:117] "RemoveContainer" containerID="6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"
Jul 16 22:01:04.160429 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:01:04.160409    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a\": container with ID starting with 6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a not found: ID does not exist" containerID="6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"
Jul 16 22:01:04.160523 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.160437    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a"} err="failed to get container status \"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a\": rpc error: code = NotFound desc = could not find container \"6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a\": container with ID starting with 6090566a6b11b91c129f13128b086e144c85fc5117d9fea206066cf8b268e35a not found: ID does not exist"
Jul 16 22:01:04.172654 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.172632    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 22:01:04.176482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:04.176449    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-dp9qx"]
Jul 16 22:01:06.011359 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:01:06.011315    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c83f31f-29a3-4f88-953b-b37eb78a3940" path="/var/lib/kubelet/pods/4c83f31f-29a3-4f88-953b-b37eb78a3940/volumes"
Jul 16 22:02:04.969298 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969264    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:04.969762 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969747    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c83f31f-29a3-4f88-953b-b37eb78a3940" containerName="manager"
Jul 16 22:02:04.969807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c83f31f-29a3-4f88-953b-b37eb78a3940" containerName="manager"
Jul 16 22:02:04.969807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e666691e-b48f-4e70-8123-7418cd485a74" containerName="cleanup"
Jul 16 22:02:04.969807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e666691e-b48f-4e70-8123-7418cd485a74" containerName="cleanup"
Jul 16 22:02:04.969902 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969865    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c83f31f-29a3-4f88-953b-b37eb78a3940" containerName="manager"
Jul 16 22:02:04.969902 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.969874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e666691e-b48f-4e70-8123-7418cd485a74" containerName="cleanup"
Jul 16 22:02:04.973187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.973171    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:04.975913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.975885    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 22:02:04.977297 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.977276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tp54f\""
Jul 16 22:02:04.977398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.977312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 22:02:04.986114 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:04.986089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:05.008222 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.008197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5lr\" (UniqueName: \"kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.008359 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.008343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.109892 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.109864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5lr\" (UniqueName: \"kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.110164 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.110144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.112407 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.112389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.118046 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.118022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5lr\" (UniqueName: \"kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr\") pod \"maas-controller-58f7bcf9f5-rv782\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") " pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.284175 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.284150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:05.422126 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.422085    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:05.422255 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:02:05.422185    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cd8f84_a30d_4535_88c6_93bd36ef07c2.slice/crio-7bad88ce91dc3f359cd1d0b6b931dca63ffca340895527e8cd48904e858728c6 WatchSource:0}: Error finding container 7bad88ce91dc3f359cd1d0b6b931dca63ffca340895527e8cd48904e858728c6: Status 404 returned error can't find the container with id 7bad88ce91dc3f359cd1d0b6b931dca63ffca340895527e8cd48904e858728c6
Jul 16 22:02:05.477699 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:05.477659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" event={"ID":"51cd8f84-a30d-4535-88c6-93bd36ef07c2","Type":"ContainerStarted","Data":"7bad88ce91dc3f359cd1d0b6b931dca63ffca340895527e8cd48904e858728c6"}
Jul 16 22:02:06.484705 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:06.484619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" event={"ID":"51cd8f84-a30d-4535-88c6-93bd36ef07c2","Type":"ContainerStarted","Data":"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"}
Jul 16 22:02:06.484705 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:06.484692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:06.503329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:06.503277    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" podStartSLOduration=1.9151389330000002 podStartE2EDuration="2.50326311s" totalImagesPullingTime="588.124177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:06.501868081 +0000 UTC m=+885.011735163" watchObservedRunningTime="2026-07-16 22:02:06.50326311 +0000 UTC m=+885.013130227"
Jul 16 22:02:17.491642 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:17.491566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:21.958437 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:21.958406    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:02:21.958943 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:21.958924    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:02:22.115254 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.115218    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:22.115448 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.115424    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" podUID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" containerName="manager" containerID="cri-o://b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e" gracePeriod=30
Jul 16 22:02:22.383923 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.383898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:22.470819 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.470788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert\") pod \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") "
Jul 16 22:02:22.470985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.470843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg5lr\" (UniqueName: \"kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr\") pod \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\" (UID: \"51cd8f84-a30d-4535-88c6-93bd36ef07c2\") "
Jul 16 22:02:22.472879 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.472848    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr" (OuterVolumeSpecName: "kube-api-access-xg5lr") pod "51cd8f84-a30d-4535-88c6-93bd36ef07c2" (UID: "51cd8f84-a30d-4535-88c6-93bd36ef07c2"). InnerVolumeSpecName "kube-api-access-xg5lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:02:22.473010 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.472875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "51cd8f84-a30d-4535-88c6-93bd36ef07c2" (UID: "51cd8f84-a30d-4535-88c6-93bd36ef07c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:02:22.566689 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.566650    2566 generic.go:356] "Generic (PLEG): container finished" podID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" containerID="b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e" exitCode=0
Jul 16 22:02:22.566853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.566703    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-rv782"
Jul 16 22:02:22.566853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.566713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" event={"ID":"51cd8f84-a30d-4535-88c6-93bd36ef07c2","Type":"ContainerDied","Data":"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"}
Jul 16 22:02:22.566853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.566798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-rv782" event={"ID":"51cd8f84-a30d-4535-88c6-93bd36ef07c2","Type":"ContainerDied","Data":"7bad88ce91dc3f359cd1d0b6b931dca63ffca340895527e8cd48904e858728c6"}
Jul 16 22:02:22.566853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.566826    2566 scope.go:117] "RemoveContainer" containerID="b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"
Jul 16 22:02:22.571686 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.571661    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51cd8f84-a30d-4535-88c6-93bd36ef07c2-webhook-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:02:22.571686 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.571686    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5lr\" (UniqueName: \"kubernetes.io/projected/51cd8f84-a30d-4535-88c6-93bd36ef07c2-kube-api-access-xg5lr\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:02:22.576519 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.576502    2566 scope.go:117] "RemoveContainer" containerID="b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"
Jul 16 22:02:22.576762 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:02:22.576738    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e\": container with ID starting with b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e not found: ID does not exist" containerID="b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"
Jul 16 22:02:22.576842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.576770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e"} err="failed to get container status \"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e\": rpc error: code = NotFound desc = could not find container \"b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e\": container with ID starting with b663dbd487f2254ada6caf521d47cccc52c404a6acbba7cfa3ad783a7d81876e not found: ID does not exist"
Jul 16 22:02:22.590023 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.590000    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:22.598764 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:22.598743    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-rv782"]
Jul 16 22:02:23.373811 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.373775    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-sxhf9"]
Jul 16 22:02:23.374287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.374208    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" containerName="manager"
Jul 16 22:02:23.374287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.374219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" containerName="manager"
Jul 16 22:02:23.374287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.374281    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" containerName="manager"
Jul 16 22:02:23.385704 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.385680    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-sxhf9"]
Jul 16 22:02:23.385835 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.385769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.388369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.388346    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 16 22:02:23.389610 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.389587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 16 22:02:23.389724 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.389617    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tp54f\""
Jul 16 22:02:23.479587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.479555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqndr\" (UniqueName: \"kubernetes.io/projected/b555e970-cac6-4667-8742-523d8be45e71-kube-api-access-zqndr\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.479762 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.479593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b555e970-cac6-4667-8742-523d8be45e71-webhook-cert\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.581003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.580972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqndr\" (UniqueName: \"kubernetes.io/projected/b555e970-cac6-4667-8742-523d8be45e71-kube-api-access-zqndr\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.581171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.581010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b555e970-cac6-4667-8742-523d8be45e71-webhook-cert\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.583485 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.583442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b555e970-cac6-4667-8742-523d8be45e71-webhook-cert\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.589290 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.589267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqndr\" (UniqueName: \"kubernetes.io/projected/b555e970-cac6-4667-8742-523d8be45e71-kube-api-access-zqndr\") pod \"maas-controller-58f7bcf9f5-sxhf9\" (UID: \"b555e970-cac6-4667-8742-523d8be45e71\") " pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:23.696742 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:23.696655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:24.011841 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.011806    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51cd8f84-a30d-4535-88c6-93bd36ef07c2" path="/var/lib/kubelet/pods/51cd8f84-a30d-4535-88c6-93bd36ef07c2/volumes"
Jul 16 22:02:24.036019 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.035996    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f7bcf9f5-sxhf9"]
Jul 16 22:02:24.038766 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:02:24.038739    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb555e970_cac6_4667_8742_523d8be45e71.slice/crio-747e7761f8520762f2b3119ccdfcf6a3b6b02fef5e1f7d2eabef68d95158f204 WatchSource:0}: Error finding container 747e7761f8520762f2b3119ccdfcf6a3b6b02fef5e1f7d2eabef68d95158f204: Status 404 returned error can't find the container with id 747e7761f8520762f2b3119ccdfcf6a3b6b02fef5e1f7d2eabef68d95158f204
Jul 16 22:02:24.580879 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.580844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9" event={"ID":"b555e970-cac6-4667-8742-523d8be45e71","Type":"ContainerStarted","Data":"9dc698df2b8c8c68ca858b411267539fad0222baa0066b9d573890f902a634a1"}
Jul 16 22:02:24.581223 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.580906    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:02:24.581223 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.580925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9" event={"ID":"b555e970-cac6-4667-8742-523d8be45e71","Type":"ContainerStarted","Data":"747e7761f8520762f2b3119ccdfcf6a3b6b02fef5e1f7d2eabef68d95158f204"}
Jul 16 22:02:24.599705 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:24.599660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9" podStartSLOduration=1.191481237 podStartE2EDuration="1.599646283s" totalImagesPullingTime="408.165046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:24.598586987 +0000 UTC m=+903.108454084" watchObservedRunningTime="2026-07-16 22:02:24.599646283 +0000 UTC m=+903.109513362"
Jul 16 22:02:35.589639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:02:35.589607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f7bcf9f5-sxhf9"
Jul 16 22:07:21.991901 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:07:21.991817    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:07:21.993265 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:07:21.993241    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:12:22.028024 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:12:22.027991    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:12:22.039093 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:12:22.039062    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:14:39.837583 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.837543    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:14:39.841418 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.841397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.844251 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.844228    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eea3cfe3-openshift-default-dockercfg-pb29p\""
Jul 16 22:14:39.875933 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.875895    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:14:39.943249 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943249 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943578 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943578 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943578 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9wn\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943578 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943760 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943760 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:39.943760 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:39.943651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9wn\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.044723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045031 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045031 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045031 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.044930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045336 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.045306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045486 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.045310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.045752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.045731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.046978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.046952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.047095 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.047078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.058607 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.058576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9wn\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.058872 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.058848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token\") pod \"e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") " pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.153656 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.153561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:40.320752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.320725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:14:40.322691 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:14:40.322659    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a7d6b5_6b98_4b98_8839_aeb82ce3befe.slice/crio-9496f5ed684138cca16fdbccbca80e69509767c6e00b42a6efd409fb0e767794 WatchSource:0}: Error finding container 9496f5ed684138cca16fdbccbca80e69509767c6e00b42a6efd409fb0e767794: Status 404 returned error can't find the container with id 9496f5ed684138cca16fdbccbca80e69509767c6e00b42a6efd409fb0e767794
Jul 16 22:14:40.324845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.324809    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:40.324967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.324879    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:40.324967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.324911    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:40.478287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.478132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerStarted","Data":"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"}
Jul 16 22:14:40.478287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.478199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerStarted","Data":"9496f5ed684138cca16fdbccbca80e69509767c6e00b42a6efd409fb0e767794"}
Jul 16 22:14:40.510039 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:40.509969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podStartSLOduration=1.509946571 podStartE2EDuration="1.509946571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:40.509923738 +0000 UTC m=+1639.019790841" watchObservedRunningTime="2026-07-16 22:14:40.509946571 +0000 UTC m=+1639.019813654"
Jul 16 22:14:41.154692 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.154660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:41.159565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.159534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:41.307385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.307343    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:14:41.311702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.311677    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.315912 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.315890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eea3cfe3-serving-cert\""
Jul 16 22:14:41.318371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.318342    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:14:41.342401 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.342349    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:14:41.458870 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.458771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.458870 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.458817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.459081 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.458891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44dj\" (UniqueName: \"kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.483134 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.483101    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:41.483981 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.483962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:41.560439 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.560408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.560639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.560449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.560639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.560517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44dj\" (UniqueName: \"kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.561187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.561162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.562960 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.562939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.575103 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.575067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44dj\" (UniqueName: \"kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj\") pod \"maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.622786 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.622754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:41.768545 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.768513    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:14:41.770204 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:14:41.770172    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fce479_9cc6_4f6d_af12_c7721681ff13.slice/crio-828dd58e428ab9adcb89b6ff242c82b93205ca78a0f6d2da00892241530797ab WatchSource:0}: Error finding container 828dd58e428ab9adcb89b6ff242c82b93205ca78a0f6d2da00892241530797ab: Status 404 returned error can't find the container with id 828dd58e428ab9adcb89b6ff242c82b93205ca78a0f6d2da00892241530797ab
Jul 16 22:14:41.771735 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:41.771717    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:14:42.489306 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:42.489266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerStarted","Data":"d0f0059e6fce9c53e285a295f5ed5aab9c8e5118d5c9846276aea0a0d89bcd79"}
Jul 16 22:14:42.489775 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:42.489331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerStarted","Data":"828dd58e428ab9adcb89b6ff242c82b93205ca78a0f6d2da00892241530797ab"}
Jul 16 22:14:42.528676 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:42.528624    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podStartSLOduration=0.980939114 podStartE2EDuration="1.5286081s" totalImagesPullingTime="547.668986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:42.52750895 +0000 UTC m=+1641.037376033" watchObservedRunningTime="2026-07-16 22:14:42.5286081 +0000 UTC m=+1641.038475181"
Jul 16 22:14:43.496066 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:43.496034    2566 generic.go:356] "Generic (PLEG): container finished" podID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerID="d0f0059e6fce9c53e285a295f5ed5aab9c8e5118d5c9846276aea0a0d89bcd79" exitCode=1
Jul 16 22:14:43.496481 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:43.496123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"d0f0059e6fce9c53e285a295f5ed5aab9c8e5118d5c9846276aea0a0d89bcd79"}
Jul 16 22:14:43.496481 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:43.496367    2566 scope.go:117] "RemoveContainer" containerID="d0f0059e6fce9c53e285a295f5ed5aab9c8e5118d5c9846276aea0a0d89bcd79"
Jul 16 22:14:44.503764 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:44.503731    2566 generic.go:356] "Generic (PLEG): container finished" podID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6" exitCode=1
Jul 16 22:14:44.504166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:44.503773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"}
Jul 16 22:14:44.504166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:44.503820    2566 scope.go:117] "RemoveContainer" containerID="d0f0059e6fce9c53e285a295f5ed5aab9c8e5118d5c9846276aea0a0d89bcd79"
Jul 16 22:14:44.504166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:44.504076    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:14:44.504332 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:44.504308    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:14:45.511583 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:45.511550    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:14:45.512053 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:45.511752    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:14:46.027209 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.027165    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:14:46.032535 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.032509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:14:46.036112 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.036089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:14:46.049551 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.049524    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:14:46.104595 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.104372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm\" (UID: \"5a1de38d-a11e-48eb-a259-9c73854f8bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:14:46.205805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.205755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm\" (UID: \"5a1de38d-a11e-48eb-a259-9c73854f8bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:14:46.206434 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.206412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm\" (UID: \"5a1de38d-a11e-48eb-a259-9c73854f8bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:14:46.344539 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.344419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:14:46.518743 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:46.518716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:14:46.520677 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:14:46.520651    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1de38d_a11e_48eb_a259_9c73854f8bf6.slice/crio-fdde2bf27ef2093eef6277132c948b5b878df93fd96e445c6386d1af74f313f5 WatchSource:0}: Error finding container fdde2bf27ef2093eef6277132c948b5b878df93fd96e445c6386d1af74f313f5: Status 404 returned error can't find the container with id fdde2bf27ef2093eef6277132c948b5b878df93fd96e445c6386d1af74f313f5
Jul 16 22:14:47.489914 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:47.489874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:47.490290 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:47.490276    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:14:47.490520 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:47.490500    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:14:47.536350 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:47.533251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerStarted","Data":"83722c7ab6699735e9b67babefcce94605e7a5e8b195495bbe550672da6f5b51"}
Jul 16 22:14:47.536350 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:47.533320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerStarted","Data":"fdde2bf27ef2093eef6277132c948b5b878df93fd96e445c6386d1af74f313f5"}
Jul 16 22:14:47.599684 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:47.599628    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" podStartSLOduration=1.599610475 podStartE2EDuration="1.599610475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:47.598704914 +0000 UTC m=+1646.108571997" watchObservedRunningTime="2026-07-16 22:14:47.599610475 +0000 UTC m=+1646.109477556"
Jul 16 22:14:48.545797 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:48.544321    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerID="83722c7ab6699735e9b67babefcce94605e7a5e8b195495bbe550672da6f5b51" exitCode=7
Jul 16 22:14:48.545797 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:48.544416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerDied","Data":"83722c7ab6699735e9b67babefcce94605e7a5e8b195495bbe550672da6f5b51"}
Jul 16 22:14:48.545797 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:48.544796    2566 scope.go:117] "RemoveContainer" containerID="83722c7ab6699735e9b67babefcce94605e7a5e8b195495bbe550672da6f5b51"
Jul 16 22:14:49.553658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:49.553625    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerID="283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9" exitCode=7
Jul 16 22:14:49.554040 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:49.553703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerDied","Data":"283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9"}
Jul 16 22:14:49.554040 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:49.553767    2566 scope.go:117] "RemoveContainer" containerID="83722c7ab6699735e9b67babefcce94605e7a5e8b195495bbe550672da6f5b51"
Jul 16 22:14:49.554219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:49.554044    2566 scope.go:117] "RemoveContainer" containerID="283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9"
Jul 16 22:14:49.554280 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:49.554243    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm_odh-ai-gateway-infra(5a1de38d-a11e-48eb-a259-9c73854f8bf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6"
Jul 16 22:14:50.561150 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:50.561073    2566 scope.go:117] "RemoveContainer" containerID="283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9"
Jul 16 22:14:50.561594 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:50.561260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm_odh-ai-gateway-infra(5a1de38d-a11e-48eb-a259-9c73854f8bf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6"
Jul 16 22:14:51.623828 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:51.623787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:14:51.624220 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:51.624208    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:14:51.624403 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:14:51.624383    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:14:53.580999 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.580965    2566 generic.go:356] "Generic (PLEG): container finished" podID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerID="92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723" exitCode=0
Jul 16 22:14:53.581420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.581042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerDied","Data":"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"}
Jul 16 22:14:53.581532 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.581517    2566 scope.go:117] "RemoveContainer" containerID="92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"
Jul 16 22:14:53.582617 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.582590    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:53.582702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.582688    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:53.582740 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:53.582726    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:14:54.588316 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:54.588280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerStarted","Data":"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"}
Jul 16 22:14:55.153682 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:55.153650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:55.154781 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:55.154753    2566 patch_prober.go:28] interesting pod/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 16 22:14:55.154909 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:55.154803    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 16 22:14:56.154440 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:56.154398    2566 patch_prober.go:28] interesting pod/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 16 22:14:56.154842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:56.154475    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 16 22:14:56.483193 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:56.483108    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:57.154154 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:57.154116    2566 patch_prober.go:28] interesting pod/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 16 22:14:57.154328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:57.154175    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 16 22:14:58.154043 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:58.154008    2566 patch_prober.go:28] interesting pod/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 16 22:14:58.154444 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:58.154068    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 16 22:14:59.158298 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:59.158266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:14:59.159298 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:14:59.159275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:15:00.140323 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.140283    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"]
Jul 16 22:15:00.144553 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.144518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"
Jul 16 22:15:00.154036 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.154016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"
Jul 16 22:15:00.158915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.158886    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"]
Jul 16 22:15:00.509474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.509432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"]
Jul 16 22:15:00.511726 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:15:00.511699    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5557b882_29fd_44a3_bd5a_9c97ebccf7f6.slice/crio-e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9 WatchSource:0}: Error finding container e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9: Status 404 returned error can't find the container with id e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9
Jul 16 22:15:00.622336 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:00.622296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr" event={"ID":"5557b882-29fd-44a3-bd5a-9c97ebccf7f6","Type":"ContainerStarted","Data":"e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9"}
Jul 16 22:15:01.005108 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:01.005078    2566 scope.go:117] "RemoveContainer" containerID="283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9"
Jul 16 22:15:01.629329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:01.629291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerStarted","Data":"ae9f308aa6254cd9865ea10c60a5e2f0ed1357c4d35eb67879b2600e6eb71de2"}
Jul 16 22:15:01.630688 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:01.630665    2566 generic.go:356] "Generic (PLEG): container finished" podID="5557b882-29fd-44a3-bd5a-9c97ebccf7f6" containerID="28becdd666466ffce41341d1af633359c69773978dd73c3e103f23126f8f1187" exitCode=0
Jul 16 22:15:01.630807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:01.630739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr" event={"ID":"5557b882-29fd-44a3-bd5a-9c97ebccf7f6","Type":"ContainerDied","Data":"28becdd666466ffce41341d1af633359c69773978dd73c3e103f23126f8f1187"}
Jul 16 22:15:02.640391 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.640351    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerID="ae9f308aa6254cd9865ea10c60a5e2f0ed1357c4d35eb67879b2600e6eb71de2" exitCode=7
Jul 16 22:15:02.640780 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.640426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerDied","Data":"ae9f308aa6254cd9865ea10c60a5e2f0ed1357c4d35eb67879b2600e6eb71de2"}
Jul 16 22:15:02.640780 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.640535    2566 scope.go:117] "RemoveContainer" containerID="283bd297a8b35313207c12521b802abee5295d65651ed0d96cde388d052e3fb9"
Jul 16 22:15:02.640932 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.640915    2566 scope.go:117] "RemoveContainer" containerID="ae9f308aa6254cd9865ea10c60a5e2f0ed1357c4d35eb67879b2600e6eb71de2"
Jul 16 22:15:02.641265 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:02.641215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm_odh-ai-gateway-infra(5a1de38d-a11e-48eb-a259-9c73854f8bf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6"
Jul 16 22:15:02.660897 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.660871    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:15:02.789881 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:02.789851    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"
Jul 16 22:15:03.649921 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.649887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr"
Jul 16 22:15:03.650252 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.649889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-fdqjr" event={"ID":"5557b882-29fd-44a3-bd5a-9c97ebccf7f6","Type":"ContainerDied","Data":"e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9"}
Jul 16 22:15:03.650252 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.650038    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d193d34e201beac1a49e91bea990eaadc60394110f139a6e3de5bde40e14b9"
Jul 16 22:15:03.803903 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.803875    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:15:03.886398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.886355    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca\") pod \"5a1de38d-a11e-48eb-a259-9c73854f8bf6\" (UID: \"5a1de38d-a11e-48eb-a259-9c73854f8bf6\") "
Jul 16 22:15:03.886732 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.886706    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a1de38d-a11e-48eb-a259-9c73854f8bf6" (UID: "5a1de38d-a11e-48eb-a259-9c73854f8bf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:15:03.988106 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:03.988020    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1de38d-a11e-48eb-a259-9c73854f8bf6-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:15:04.656246 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:04.656212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"
Jul 16 22:15:04.656246 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:04.656242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm" event={"ID":"5a1de38d-a11e-48eb-a259-9c73854f8bf6","Type":"ContainerDied","Data":"fdde2bf27ef2093eef6277132c948b5b878df93fd96e445c6386d1af74f313f5"}
Jul 16 22:15:04.656817 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:04.656303    2566 scope.go:117] "RemoveContainer" containerID="ae9f308aa6254cd9865ea10c60a5e2f0ed1357c4d35eb67879b2600e6eb71de2"
Jul 16 22:15:04.675188 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:04.675156    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:15:04.680823 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:04.680801    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-424hm"]
Jul 16 22:15:06.005451 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:06.005420    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:15:06.012326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:06.012300    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" path="/var/lib/kubelet/pods/5a1de38d-a11e-48eb-a259-9c73854f8bf6/volumes"
Jul 16 22:15:06.672802 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:06.672769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerStarted","Data":"10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"}
Jul 16 22:15:06.673190 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:06.673172    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:06.673391 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:06.673362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:07.490028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:07.489980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:15:07.679841 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:07.679798    2566 generic.go:356] "Generic (PLEG): container finished" podID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921" exitCode=1
Jul 16 22:15:07.680039 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:07.679891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"}
Jul 16 22:15:07.680039 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:07.679964    2566 scope.go:117] "RemoveContainer" containerID="b714ed9c6c94f5816fe5b0b710909de9d5fea4806d9aa93c85e6b8ed1f00a4f6"
Jul 16 22:15:07.680276 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:07.680255    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:07.680516 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:07.680492    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:08.688120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:08.688091    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:08.688535 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:08.688303    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:11.623965 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:11.623929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:15:11.624386 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:11.624349    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:11.624570 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:11.624550    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:25.005117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:25.005087    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:25.005556 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:25.005257    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:38.005105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:38.005070    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:38.860360 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:38.860329    2566 generic.go:356] "Generic (PLEG): container finished" podID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d" exitCode=1
Jul 16 22:15:38.860546 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:38.860374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"}
Jul 16 22:15:38.860546 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:38.860443    2566 scope.go:117] "RemoveContainer" containerID="10037756f349efff6eaa3ba3bdcdbf14204797c031c27ea167cfcb7571898921"
Jul 16 22:15:38.860792 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:38.860772    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:15:38.861048 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:38.861018    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:41.623938 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:41.623908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:15:41.624317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:41.624282    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:15:41.624516 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:41.624485    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:42.490055 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:42.490011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:15:42.490474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:42.490440    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:15:42.490654 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:42.490633    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:15:54.004877 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:15:54.004842    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:15:54.005488 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:15:54.005039    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:06.005223 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:06.005193    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:16:06.005623 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:06.005382    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:17.004812 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:17.004776    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:16:17.005223 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:17.004988    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:30.005524 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:30.005450    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:16:31.162228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:31.162142    2566 generic.go:356] "Generic (PLEG): container finished" podID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581" exitCode=1
Jul 16 22:16:31.162228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:31.162210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"}
Jul 16 22:16:31.162693 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:31.162301    2566 scope.go:117] "RemoveContainer" containerID="d4623ebb3dd628bb46e37b36b1fc80c7407757c370eaaa2d5477d7227d6f476d"
Jul 16 22:16:31.162693 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:31.162670    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:16:31.162910 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:31.162889    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:31.623601 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:31.623565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:16:32.169397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:32.169358    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:16:32.169800 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:32.169579    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:32.489426 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:32.489398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:16:33.174760 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:33.174732    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:16:33.175126 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:33.174927    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:16:46.011564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:16:46.011526    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:16:46.012022 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:16:46.011811    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:17:01.005105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:01.005074    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:17:01.005545 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:01.005293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:17:13.004896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:13.004857    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:17:13.005486 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:13.005130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:17:22.070303 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:22.070255    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:17:22.074850 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:22.074828    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:17:24.005071 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:24.005041    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:17:24.005445 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:24.005237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:17:38.005102 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:38.005066    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:17:38.005533 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:38.005257    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss_odh-ai-gateway-infra(20fce479-9cc6-4f6d-af12-c7721681ff13)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13"
Jul 16 22:17:46.131802 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:46.131766    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:17:46.132171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:46.131977    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy" containerID="cri-o://7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179" gracePeriod=30
Jul 16 22:17:51.396824 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.396755    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:17:51.512133 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512133 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512199    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512223    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9wn\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512412    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy\") pod \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\" (UID: \"30a7d6b5-6b98-4b98-8839-aeb82ce3befe\") "
Jul 16 22:17:51.512821 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:51.512890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:51.512890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512586    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:51.512890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.512820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:51.513169 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.513134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data" (OuterVolumeSpecName: "istio-data") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:51.514620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.514597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:17:51.514717 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.514622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:51.514925 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.514901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn" (OuterVolumeSpecName: "kube-api-access-2h9wn") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "kube-api-access-2h9wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:51.514985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.514936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token" (OuterVolumeSpecName: "istio-token") pod "30a7d6b5-6b98-4b98-8839-aeb82ce3befe" (UID: "30a7d6b5-6b98-4b98-8839-aeb82ce3befe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:51.613853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613821    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.613853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613851    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9wn\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-kube-api-access-2h9wn\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.613853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613862    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613872    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613880    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613887    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613895    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613902    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.614117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.613914    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a7d6b5-6b98-4b98-8839-aeb82ce3befe-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:51.620968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.620935    2566 generic.go:356] "Generic (PLEG): container finished" podID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerID="7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179" exitCode=0
Jul 16 22:17:51.621094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.620995    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"
Jul 16 22:17:51.621094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.621010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerDied","Data":"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"}
Jul 16 22:17:51.621094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.621066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk" event={"ID":"30a7d6b5-6b98-4b98-8839-aeb82ce3befe","Type":"ContainerDied","Data":"9496f5ed684138cca16fdbccbca80e69509767c6e00b42a6efd409fb0e767794"}
Jul 16 22:17:51.621094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.621088    2566 scope.go:117] "RemoveContainer" containerID="7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"
Jul 16 22:17:51.630765 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.630739    2566 scope.go:117] "RemoveContainer" containerID="92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"
Jul 16 22:17:51.639101 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.639083    2566 scope.go:117] "RemoveContainer" containerID="7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"
Jul 16 22:17:51.639362 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:51.639343    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179\": container with ID starting with 7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179 not found: ID does not exist" containerID="7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"
Jul 16 22:17:51.639411 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.639371    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179"} err="failed to get container status \"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179\": rpc error: code = NotFound desc = could not find container \"7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179\": container with ID starting with 7052130732daf1b494405d0ae645ff62844ce9fc49ed62615823a247a564b179 not found: ID does not exist"
Jul 16 22:17:51.639411 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.639388    2566 scope.go:117] "RemoveContainer" containerID="92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"
Jul 16 22:17:51.639659 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:17:51.639641    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723\": container with ID starting with 92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723 not found: ID does not exist" containerID="92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"
Jul 16 22:17:51.639706 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.639664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723"} err="failed to get container status \"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723\": rpc error: code = NotFound desc = could not find container \"92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723\": container with ID starting with 92d93b213674608f68fc6e3bda0b7ea409d5ad5cf4ef0ba8aef17962a42fa723 not found: ID does not exist"
Jul 16 22:17:51.643751 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.643727    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:17:51.648428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.648367    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eea3cfe3-openshift-default-566d69c87c-qvqlk"]
Jul 16 22:17:51.881823 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:51.881782    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:17:52.013692 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.013666    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" path="/var/lib/kubelet/pods/30a7d6b5-6b98-4b98-8839-aeb82ce3befe/volumes"
Jul 16 22:17:52.024148 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.024120    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:17:52.118369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.118334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca\") pod \"20fce479-9cc6-4f6d-af12-c7721681ff13\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") "
Jul 16 22:17:52.118570 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.118427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls\") pod \"20fce479-9cc6-4f6d-af12-c7721681ff13\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") "
Jul 16 22:17:52.118570 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.118522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p44dj\" (UniqueName: \"kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj\") pod \"20fce479-9cc6-4f6d-af12-c7721681ff13\" (UID: \"20fce479-9cc6-4f6d-af12-c7721681ff13\") "
Jul 16 22:17:52.118807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.118775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20fce479-9cc6-4f6d-af12-c7721681ff13" (UID: "20fce479-9cc6-4f6d-af12-c7721681ff13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:52.120614 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.120587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20fce479-9cc6-4f6d-af12-c7721681ff13" (UID: "20fce479-9cc6-4f6d-af12-c7721681ff13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:17:52.120725 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.120620    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj" (OuterVolumeSpecName: "kube-api-access-p44dj") pod "20fce479-9cc6-4f6d-af12-c7721681ff13" (UID: "20fce479-9cc6-4f6d-af12-c7721681ff13"). InnerVolumeSpecName "kube-api-access-p44dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:52.219869 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.219792    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44dj\" (UniqueName: \"kubernetes.io/projected/20fce479-9cc6-4f6d-af12-c7721681ff13-kube-api-access-p44dj\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:52.219869 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.219819    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fce479-9cc6-4f6d-af12-c7721681ff13-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:52.219869 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.219833    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fce479-9cc6-4f6d-af12-c7721681ff13-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:17:52.628732 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.628698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss" event={"ID":"20fce479-9cc6-4f6d-af12-c7721681ff13","Type":"ContainerDied","Data":"828dd58e428ab9adcb89b6ff242c82b93205ca78a0f6d2da00892241530797ab"}
Jul 16 22:17:52.628732 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.628731    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"
Jul 16 22:17:52.629169 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.628758    2566 scope.go:117] "RemoveContainer" containerID="141630cc33611b56980a96ab4fdf160cb8033e5b46f8d10ab4dfa96c0b5b7581"
Jul 16 22:17:52.652194 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.652168    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:17:52.657020 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:52.656995    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eea3cfe3-778f78979d-fq8ss"]
Jul 16 22:17:54.010907 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:54.010865    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" path="/var/lib/kubelet/pods/20fce479-9cc6-4f6d-af12-c7721681ff13/volumes"
Jul 16 22:17:58.737789 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.737759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:17:58.738194 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738182    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738195    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738214    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738220    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738228    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738242    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738247    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738253    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738268    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738273    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738295    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738307    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738313    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738319    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5557b882-29fd-44a3-bd5a-9c97ebccf7f6" containerName="cleanup"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738323    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5557b882-29fd-44a3-bd5a-9c97ebccf7f6" containerName="cleanup"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738394    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738401    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738406    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738411    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a7d6b5-6b98-4b98-8839-aeb82ce3befe" containerName="istio-proxy"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738418    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738425    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5557b882-29fd-44a3-bd5a-9c97ebccf7f6" containerName="cleanup"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738435    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.738493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.738441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:17:58.741670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.741654    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.744796 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.744778    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f123e265-gw-openshift-default-dockercfg-6b85q\""
Jul 16 22:17:58.779894 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.779866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:17:58.880142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880316 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880316 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpzv\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880316 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880316 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.880678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.880535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981711 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981711 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981711 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpzv\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.981984 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.981908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.982271 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.982092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.982271 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.982142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.982384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.982306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.982384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.982356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.982518 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.982496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.984112 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.984091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.984328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.984312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.990295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.990273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:58.990394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:58.990313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpzv\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv\") pod \"e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") " pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:59.052403 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.052366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:17:59.196373 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.196342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:17:59.198118 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:17:59.198091    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab25ed2_2c14_4dd4_b7cb_8e3e2882a86a.slice/crio-900424b5a21e4ff2c40c3c2f9eacefd36f5c47fde37efd7848a41812c0046c26 WatchSource:0}: Error finding container 900424b5a21e4ff2c40c3c2f9eacefd36f5c47fde37efd7848a41812c0046c26: Status 404 returned error can't find the container with id 900424b5a21e4ff2c40c3c2f9eacefd36f5c47fde37efd7848a41812c0046c26
Jul 16 22:17:59.200295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.200266    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:17:59.200361 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.200348    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:17:59.200401 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.200381    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:17:59.668414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.668321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerStarted","Data":"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"}
Jul 16 22:17:59.668414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.668374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerStarted","Data":"900424b5a21e4ff2c40c3c2f9eacefd36f5c47fde37efd7848a41812c0046c26"}
Jul 16 22:17:59.689916 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:17:59.689870    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podStartSLOduration=1.68985626 podStartE2EDuration="1.68985626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:59.688425542 +0000 UTC m=+1838.198292626" watchObservedRunningTime="2026-07-16 22:17:59.68985626 +0000 UTC m=+1838.199723342"
Jul 16 22:18:00.053180 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.053138    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:00.058709 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.058683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:00.122470 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.122422    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:18:00.122965 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.122945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:18:00.123058 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.122967    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:18:00.123120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.123081    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a1de38d-a11e-48eb-a259-9c73854f8bf6" containerName="revoke-keys"
Jul 16 22:18:00.123120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.123098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fce479-9cc6-4f6d-af12-c7721681ff13" containerName="maas-api"
Jul 16 22:18:00.126277 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.126253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.131227 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.131199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f123e265-serving-cert\""
Jul 16 22:18:00.131350 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.131321    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:18:00.142446 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.142423    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:18:00.192896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.192865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46s8b\" (UniqueName: \"kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.193018 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.192902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.193073 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.193020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.293598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.293557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46s8b\" (UniqueName: \"kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.293775 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.293605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.293775 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.293709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.294575 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.294547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.296281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.296248    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.302841 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.302815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46s8b\" (UniqueName: \"kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b\") pod \"maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.439322 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.439233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:00.584990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.584960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:18:00.586568 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:18:00.586533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2305e3_6357_4a7f_bbd4_84fd89187aee.slice/crio-9de5e21295d4f83e93fcb469785d2e74d01e21d4f484f43cc8c5f8db3b60a94a WatchSource:0}: Error finding container 9de5e21295d4f83e93fcb469785d2e74d01e21d4f484f43cc8c5f8db3b60a94a: Status 404 returned error can't find the container with id 9de5e21295d4f83e93fcb469785d2e74d01e21d4f484f43cc8c5f8db3b60a94a
Jul 16 22:18:00.675331 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.675289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerStarted","Data":"9de5e21295d4f83e93fcb469785d2e74d01e21d4f484f43cc8c5f8db3b60a94a"}
Jul 16 22:18:00.675648 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.675621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:00.676665 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:00.676646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:02.687089 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:02.687048    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerID="7eeb2b0553e8ae3de6332d43b9bb66defb1aa21951527f31907d8f90dff86024" exitCode=1
Jul 16 22:18:02.687516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:02.687140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerDied","Data":"7eeb2b0553e8ae3de6332d43b9bb66defb1aa21951527f31907d8f90dff86024"}
Jul 16 22:18:02.687516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:02.687421    2566 scope.go:117] "RemoveContainer" containerID="7eeb2b0553e8ae3de6332d43b9bb66defb1aa21951527f31907d8f90dff86024"
Jul 16 22:18:03.694270 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:03.694232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerStarted","Data":"d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"}
Jul 16 22:18:03.694766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:03.694330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:03.716028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:03.715973    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podStartSLOduration=1.626473028 podStartE2EDuration="3.715958903s" totalImagesPullingTime="2.089485875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:03.713720678 +0000 UTC m=+1842.223587763" watchObservedRunningTime="2026-07-16 22:18:03.715958903 +0000 UTC m=+1842.225825984"
Jul 16 22:18:04.701511 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:04.701473    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea" exitCode=1
Jul 16 22:18:04.701916 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:04.701516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerDied","Data":"d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"}
Jul 16 22:18:04.701916 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:04.701581    2566 scope.go:117] "RemoveContainer" containerID="7eeb2b0553e8ae3de6332d43b9bb66defb1aa21951527f31907d8f90dff86024"
Jul 16 22:18:04.701916 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:04.701876    2566 scope.go:117] "RemoveContainer" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"
Jul 16 22:18:04.702120 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:04.702094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:05.716437 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:05.716404    2566 scope.go:117] "RemoveContainer" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"
Jul 16 22:18:05.716844 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:05.716618    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:10.439384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.439355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:10.446718 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.439732    2566 scope.go:117] "RemoveContainer" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"
Jul 16 22:18:10.446718 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:10.439929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:10.749046 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.748994    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerID="1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a" exitCode=0
Jul 16 22:18:10.749267 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.749182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerDied","Data":"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"}
Jul 16 22:18:10.749825 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.749801    2566 scope.go:117] "RemoveContainer" containerID="1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"
Jul 16 22:18:10.751431 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.751388    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:18:10.751583 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.751562    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:18:10.751694 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:10.751676    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:18:11.756990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:11.756945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerStarted","Data":"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"}
Jul 16 22:18:12.053196 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:12.053103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:12.054397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:12.054372    2566 patch_prober.go:28] interesting pod/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 16 22:18:12.054480 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:12.054426    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 16 22:18:13.053274 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:13.053230    2566 patch_prober.go:28] interesting pod/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 16 22:18:13.053719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:13.053309    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 16 22:18:14.052953 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:14.052915    2566 patch_prober.go:28] interesting pod/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 16 22:18:14.053131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:14.052970    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 16 22:18:15.053615 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:15.053578    2566 patch_prober.go:28] interesting pod/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Jul 16 22:18:15.053991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:15.053637    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Jul 16 22:18:15.675753 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:15.675714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:16.057131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:16.057102    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:16.058298 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:16.058276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:18:21.004503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:21.004441    2566 scope.go:117] "RemoveContainer" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"
Jul 16 22:18:21.814652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:21.814620    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05" exitCode=1
Jul 16 22:18:21.814821 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:21.814693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerDied","Data":"07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"}
Jul 16 22:18:21.814821 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:21.814759    2566 scope.go:117] "RemoveContainer" containerID="d853d8eb5e58a446e01a39b3f3245dffacfe90311d3a848c50cb1ee540f162ea"
Jul 16 22:18:21.815136 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:21.815117    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:21.815367 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:21.815343    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:23.694710 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:23.694672    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:23.695099 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:23.695035    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:23.695255 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:23.695235    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:30.439813 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:30.439774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:30.440235 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:30.440214    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:30.440564 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:30.440535    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:41.004787 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:41.004753    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:41.005143 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:41.004929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:52.007866 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:52.007835    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:52.986395 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:52.986355    2566 generic.go:356] "Generic (PLEG): container finished" podID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerID="4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04" exitCode=1
Jul 16 22:18:52.986666 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:52.986413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerDied","Data":"4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04"}
Jul 16 22:18:52.986666 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:52.986519    2566 scope.go:117] "RemoveContainer" containerID="07c2a074a9d4a5035021ee69677b7fcf690978e8e9515c543a42605265f7ae05"
Jul 16 22:18:52.986939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:52.986906    2566 scope.go:117] "RemoveContainer" containerID="4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04"
Jul 16 22:18:52.987186 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:52.987164    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:18:53.695449 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:53.695413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:18:53.993315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:18:53.993276    2566 scope.go:117] "RemoveContainer" containerID="4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04"
Jul 16 22:18:53.993503 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:18:53.993480    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:19:00.439995 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:00.439957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:19:00.440412 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:00.440400    2566 scope.go:117] "RemoveContainer" containerID="4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04"
Jul 16 22:19:00.440630 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:00.440608    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f_odh-ai-gateway-infra(ab2305e3-6357-4a7f-bbd4-84fd89187aee)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee"
Jul 16 22:19:04.643923 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:04.643879    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:19:04.644384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:04.644149    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy" containerID="cri-o://06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc" gracePeriod=30
Jul 16 22:19:09.912382 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.912321    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:19:09.995555 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995442    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995653    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995728    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqpzv\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.995951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.995883    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert\") pod \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\" (UID: \"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a\") "
Jul 16 22:19:09.996171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:09.996667 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996338    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:09.996667 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:09.996667 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996614    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:09.996908 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:09.997020 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.996985    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:09.997952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.997925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:09.998181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.998157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:19:09.998282 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.998156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:09.998583 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:09.998567    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv" (OuterVolumeSpecName: "kube-api-access-hqpzv") pod "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" (UID: "5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a"). InnerVolumeSpecName "kube-api-access-hqpzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:10.080494 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.080443    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerID="06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc" exitCode=0
Jul 16 22:19:10.080643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.080528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerDied","Data":"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"}
Jul 16 22:19:10.080643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.080583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd" event={"ID":"5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a","Type":"ContainerDied","Data":"900424b5a21e4ff2c40c3c2f9eacefd36f5c47fde37efd7848a41812c0046c26"}
Jul 16 22:19:10.080643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.080539    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"
Jul 16 22:19:10.080802 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.080610    2566 scope.go:117] "RemoveContainer" containerID="06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"
Jul 16 22:19:10.090049 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.090031    2566 scope.go:117] "RemoveContainer" containerID="1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"
Jul 16 22:19:10.097009 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.096986    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097009 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097009    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097020    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097028    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097037    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097049    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097060    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097068    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqpzv\" (UniqueName: \"kubernetes.io/projected/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a-kube-api-access-hqpzv\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.097804 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.097793    2566 scope.go:117] "RemoveContainer" containerID="06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"
Jul 16 22:19:10.098078 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:10.098057    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc\": container with ID starting with 06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc not found: ID does not exist" containerID="06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"
Jul 16 22:19:10.098151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.098087    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc"} err="failed to get container status \"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc\": rpc error: code = NotFound desc = could not find container \"06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc\": container with ID starting with 06b2df7a811b64b86831e790b392c3c84ba753b742ec2b94970f5d7d09bf97dc not found: ID does not exist"
Jul 16 22:19:10.098151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.098105    2566 scope.go:117] "RemoveContainer" containerID="1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"
Jul 16 22:19:10.098334 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:10.098318    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a\": container with ID starting with 1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a not found: ID does not exist" containerID="1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"
Jul 16 22:19:10.098384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.098338    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a"} err="failed to get container status \"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a\": rpc error: code = NotFound desc = could not find container \"1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a\": container with ID starting with 1280c514653d84ab46e8900f215fea51c6b94a3c57513421343763e6ef3cfc9a not found: ID does not exist"
Jul 16 22:19:10.123162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.123132    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:19:10.127182 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.127153    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f123e265-gw-openshift-default-77f8cc497f-p9jnd"]
Jul 16 22:19:10.512925 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.512888    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:19:10.628411 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.628385    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:19:10.702642 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.702609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls\") pod \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") "
Jul 16 22:19:10.702808 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.702719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46s8b\" (UniqueName: \"kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b\") pod \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") "
Jul 16 22:19:10.702808 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.702761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca\") pod \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\" (UID: \"ab2305e3-6357-4a7f-bbd4-84fd89187aee\") "
Jul 16 22:19:10.703145 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.703121    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab2305e3-6357-4a7f-bbd4-84fd89187aee" (UID: "ab2305e3-6357-4a7f-bbd4-84fd89187aee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:10.704631 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.704607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab2305e3-6357-4a7f-bbd4-84fd89187aee" (UID: "ab2305e3-6357-4a7f-bbd4-84fd89187aee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:10.704740 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.704719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b" (OuterVolumeSpecName: "kube-api-access-46s8b") pod "ab2305e3-6357-4a7f-bbd4-84fd89187aee" (UID: "ab2305e3-6357-4a7f-bbd4-84fd89187aee"). InnerVolumeSpecName "kube-api-access-46s8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:10.804049 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.803965    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2305e3-6357-4a7f-bbd4-84fd89187aee-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.804049 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.803996    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2305e3-6357-4a7f-bbd4-84fd89187aee-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:10.804049 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:10.804006    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46s8b\" (UniqueName: \"kubernetes.io/projected/ab2305e3-6357-4a7f-bbd4-84fd89187aee-kube-api-access-46s8b\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:19:11.087375 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:11.087287    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"
Jul 16 22:19:11.087375 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:11.087306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f" event={"ID":"ab2305e3-6357-4a7f-bbd4-84fd89187aee","Type":"ContainerDied","Data":"9de5e21295d4f83e93fcb469785d2e74d01e21d4f484f43cc8c5f8db3b60a94a"}
Jul 16 22:19:11.087375 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:11.087372    2566 scope.go:117] "RemoveContainer" containerID="4420e219d295050e6162e5f66f2ac3d02de58c9c8f17013deb568e8e49dc9f04"
Jul 16 22:19:11.117422 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:11.117380    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:19:11.121975 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:11.121953    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f123e265-6b6668599b-4mx4f"]
Jul 16 22:19:12.011328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:12.011285    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" path="/var/lib/kubelet/pods/5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a/volumes"
Jul 16 22:19:12.011852 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:12.011816    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" path="/var/lib/kubelet/pods/ab2305e3-6357-4a7f-bbd4-84fd89187aee/volumes"
Jul 16 22:19:18.642333 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642303    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:19:18.642819 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.642819 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642821    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642841    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642847    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642877    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.642895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642882    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642955    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642963    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642978    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642984    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ab25ed2-2c14-4dd4-b7cb-8e3e2882a86a" containerName="istio-proxy"
Jul 16 22:19:18.643105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.642989    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:18.647635 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.647611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.652880 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.652860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f2f32f5-openshift-default-dockercfg-77bnw\""
Jul 16 22:19:18.666326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.666303    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:19:18.782785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.782746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.782785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.782786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhkn\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.782986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.782887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.782986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.782933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.782986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.782982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.783084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.783011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.783084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.783042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.783084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.783062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.783175 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.783084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883635 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883881 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883881 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883957 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.883991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.883973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884038 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhkn\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884145 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884205 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884249 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884332 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.884550 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.884510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.886392 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.886371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.886553 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.886537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.895301 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.895245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.896414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.896394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhkn\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn\") pod \"e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") " pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:18.958536 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:18.958500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:19.123923 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:19.123890    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:19:19.125198 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:19:19.125171    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f88309_588c_46ae_bfdb_0ee9f9ec8435.slice/crio-85ef50166030a8246406f8fb5b7b7686e0d084a25e207b308d78379741230dd1 WatchSource:0}: Error finding container 85ef50166030a8246406f8fb5b7b7686e0d084a25e207b308d78379741230dd1: Status 404 returned error can't find the container with id 85ef50166030a8246406f8fb5b7b7686e0d084a25e207b308d78379741230dd1
Jul 16 22:19:19.127175 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:19.127145    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:19.127279 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:19.127216    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:19.127279 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:19.127258    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:19.134235 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:19.134209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerStarted","Data":"85ef50166030a8246406f8fb5b7b7686e0d084a25e207b308d78379741230dd1"}
Jul 16 22:19:20.141969 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.141934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerStarted","Data":"7783b82565ade2c12feda8c6b0aae6e70bb72f9d33fd696776302beae9e1cbfe"}
Jul 16 22:19:20.235824 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.235787    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:19:20.236234 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.236222    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:20.236273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.236236    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:20.236273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.236254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:20.236273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.236260    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2305e3-6357-4a7f-bbd4-84fd89187aee" containerName="maas-api"
Jul 16 22:19:20.239635 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.239618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.245702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.245683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:19:20.250398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.250375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f2f32f5-serving-cert\""
Jul 16 22:19:20.252003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.251960    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podStartSLOduration=2.251948049 podStartE2EDuration="2.251948049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:20.250519941 +0000 UTC m=+1918.760387023" watchObservedRunningTime="2026-07-16 22:19:20.251948049 +0000 UTC m=+1918.761815131"
Jul 16 22:19:20.262769 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.262748    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:19:20.400552 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.400431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.400716 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.400597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.400716 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.400634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2glv\" (UniqueName: \"kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.501943 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.501907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.502109 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.502008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.502109 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.502040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2glv\" (UniqueName: \"kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.502236 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:20.502145    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-serving-cert: secret "maas-api-e2e-ait-6f2f32f5-serving-cert" not found
Jul 16 22:19:20.502236 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:20.502209    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls podName:b87cb34a-a29a-467b-af80-46275ddbad60 nodeName:}" failed. No retries permitted until 2026-07-16 22:19:21.002192802 +0000 UTC m=+1919.512059862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls") pod "maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" (UID: "b87cb34a-a29a-467b-af80-46275ddbad60") : secret "maas-api-e2e-ait-6f2f32f5-serving-cert" not found
Jul 16 22:19:20.502617 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.502596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.521258 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.521237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2glv\" (UniqueName: \"kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:20.958739 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.958699    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:20.964339 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:20.964311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:21.005326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.005292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:21.007743 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.007720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") pod \"maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:21.148225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.148195    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:21.149233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.149212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:21.150307 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.150289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:21.531778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:21.531751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:19:21.533791 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:19:21.533762    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87cb34a_a29a_467b_af80_46275ddbad60.slice/crio-54fe255333688d8c6b2ca06dffab56f54b56b4582eeb6d97572c16bc75a11ba0 WatchSource:0}: Error finding container 54fe255333688d8c6b2ca06dffab56f54b56b4582eeb6d97572c16bc75a11ba0: Status 404 returned error can't find the container with id 54fe255333688d8c6b2ca06dffab56f54b56b4582eeb6d97572c16bc75a11ba0
Jul 16 22:19:22.155660 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:22.155613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerStarted","Data":"87c687fb9df4be202ea5258bba631d11f97345d33c593e1a1cb3b150f97cd4b1"}
Jul 16 22:19:22.156084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:22.155679    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerStarted","Data":"54fe255333688d8c6b2ca06dffab56f54b56b4582eeb6d97572c16bc75a11ba0"}
Jul 16 22:19:22.156084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:22.155778    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:22.213218 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:22.213164    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podStartSLOduration=1.817274118 podStartE2EDuration="2.213143913s" totalImagesPullingTime="395.869795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:22.209532567 +0000 UTC m=+1920.719399650" watchObservedRunningTime="2026-07-16 22:19:22.213143913 +0000 UTC m=+1920.723010995"
Jul 16 22:19:23.162803 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:23.162769    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="87c687fb9df4be202ea5258bba631d11f97345d33c593e1a1cb3b150f97cd4b1" exitCode=1
Jul 16 22:19:23.163192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:23.162846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"87c687fb9df4be202ea5258bba631d11f97345d33c593e1a1cb3b150f97cd4b1"}
Jul 16 22:19:23.163192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:23.163100    2566 scope.go:117] "RemoveContainer" containerID="87c687fb9df4be202ea5258bba631d11f97345d33c593e1a1cb3b150f97cd4b1"
Jul 16 22:19:24.169805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:24.169714    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c" exitCode=1
Jul 16 22:19:24.170308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:24.169801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"}
Jul 16 22:19:24.170308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:24.169863    2566 scope.go:117] "RemoveContainer" containerID="87c687fb9df4be202ea5258bba631d11f97345d33c593e1a1cb3b150f97cd4b1"
Jul 16 22:19:24.170308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:24.170002    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:24.170308 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:24.170233    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:25.177225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:25.177193    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:25.177669 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:25.177447    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:27.156022 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:27.155988    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:27.156435 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:27.156419    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:27.156650 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:27.156630    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:31.151435 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:31.151392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:31.151881 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:31.151865    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:31.152105 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:31.152084    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:37.188634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:37.188584    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 22:19:37.189022 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:37.188668    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:19:38.248814 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.248777    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerID="7783b82565ade2c12feda8c6b0aae6e70bb72f9d33fd696776302beae9e1cbfe" exitCode=0
Jul 16 22:19:38.249299 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.248828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerDied","Data":"7783b82565ade2c12feda8c6b0aae6e70bb72f9d33fd696776302beae9e1cbfe"}
Jul 16 22:19:38.249299 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.249211    2566 scope.go:117] "RemoveContainer" containerID="7783b82565ade2c12feda8c6b0aae6e70bb72f9d33fd696776302beae9e1cbfe"
Jul 16 22:19:38.250181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.250154    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:38.250233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.250225    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:38.250269 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:38.250257    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:19:39.257807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:39.257771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerStarted","Data":"420edee084d8374dc6f5ae6850ae454f541c7cacb48ff8abdbba17bcbe63d47d"}
Jul 16 22:19:39.959121 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:39.959063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:39.959311 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:39.959286    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 16 22:19:39.959371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:39.959350    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 16 22:19:40.959756 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:40.959715    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 16 22:19:40.960144 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:40.959774    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 16 22:19:41.959571 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:41.959533    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 16 22:19:41.959754 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:41.959589    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 16 22:19:42.007371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:42.007343    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:42.007743 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:42.007645    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:19:42.958990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:42.958950    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Jul 16 22:19:42.959160 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:42.959009    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Jul 16 22:19:43.282748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.282714    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972" exitCode=1
Jul 16 22:19:43.283187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.282790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"}
Jul 16 22:19:43.283187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.282851    2566 scope.go:117] "RemoveContainer" containerID="7da94988fba9cf001738e45c9a060bf7d1b3091edc5f4acda931774cce2eef2c"
Jul 16 22:19:43.283317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.283275    2566 scope.go:117] "RemoveContainer" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"
Jul 16 22:19:43.283541 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:43.283517    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:43.963374 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.963344    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:43.963612 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.963591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:43.964389 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:43.964369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:19:47.156386 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:47.156354    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:47.156800 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:47.156765    2566 scope.go:117] "RemoveContainer" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"
Jul 16 22:19:47.156969 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:47.156947    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:19:51.151012 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:51.150984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:19:51.151346 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:19:51.151327    2566 scope.go:117] "RemoveContainer" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"
Jul 16 22:19:51.151560 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:19:51.151537    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:03.005027 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:03.004995    2566 scope.go:117] "RemoveContainer" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"
Jul 16 22:20:04.401932 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:04.401833    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f" exitCode=1
Jul 16 22:20:04.401932 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:04.401910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"}
Jul 16 22:20:04.402438 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:04.401984    2566 scope.go:117] "RemoveContainer" containerID="a386ce2962815311d745b94753feee8b533edbf935787c49bbdbbc99769ee972"
Jul 16 22:20:04.402438 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:04.402370    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:04.402686 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:04.402655    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:07.156331 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:07.156300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:20:07.156778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:07.156653    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:07.156867 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:07.156846    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:11.151202 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:11.151164    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:20:11.151772 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:11.151754    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:11.152009 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:11.151986    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:23.005440 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:23.005349    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:23.005909 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:23.005579    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:34.005402 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:34.005362    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:34.005920 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:34.005585    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:49.005122 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:49.005083    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:49.649866 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:49.649824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerStarted","Data":"c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"}
Jul 16 22:20:49.650074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:49.650053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:20:50.657827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:50.657795    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c" exitCode=1
Jul 16 22:20:50.658247 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:50.657872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"}
Jul 16 22:20:50.658247 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:50.657926    2566 scope.go:117] "RemoveContainer" containerID="0b8d6b9f7083a0d1222a5ab3d9b06abd2bbc3c5bef6b5cbddfa67ff2f0d9409f"
Jul 16 22:20:50.658247 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:50.658188    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:20:50.658487 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:50.658441    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:51.150890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:51.150861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:20:51.664802 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:51.664770    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:20:51.665183 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:51.665013    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:20:52.669842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:20:52.669811    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:20:52.670217 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:20:52.669987    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:21:04.005260 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:21:04.005222    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:21:04.005765 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:21:04.005492    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:21:18.005504 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:21:18.005435    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:21:18.006002 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:21:18.005684    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:21:32.011074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:21:32.011040    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:21:32.012006 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:21:32.011973    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:21:43.004973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:21:43.004945    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:21:43.005381 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:21:43.005130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:21:54.005390 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:21:54.005354    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:21:54.005884 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:21:54.005594    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:06.005371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:06.005342    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:22:06.005856 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:06.005561    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:18.004631 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:18.004596    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:22:19.162798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:19.162709    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e" exitCode=1
Jul 16 22:22:19.162798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:19.162785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"}
Jul 16 22:22:19.163331 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:19.162857    2566 scope.go:117] "RemoveContainer" containerID="c7b9dda6394821809cf1f67f622b3027cbd8505a462cf717d1011df41f87158c"
Jul 16 22:22:19.163331 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:19.163229    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:22:19.163514 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:19.163496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:19.650397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:19.650360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:22:20.169385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:20.169354    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:22:20.169783 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:20.169555    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:21.150560 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:21.150531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:22:21.175038 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:21.175019    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:22:21.175353 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:21.175194    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:22.105740 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:22.105710    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:22:22.111151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:22.111129    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:22:35.004994 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:35.004952    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:22:35.005491 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:35.005196    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:22:47.005472 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:22:47.005422    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:22:47.005945 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:22:47.005661    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:23:01.005037 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:23:01.005004    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:23:01.005427 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:23:01.005244    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:23:14.004944 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:23:14.004912    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:23:14.005333 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:23:14.005108    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:23:25.004871 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:23:25.004833    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:23:25.005426 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:23:25.005063    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:23:36.004990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:23:36.004957    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:23:36.005483 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:23:36.005249    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:23:51.004918 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:23:51.004886    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:23:51.005336 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:23:51.005069    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:24:04.005266 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:24:04.005228    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:24:04.005843 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:24:04.005497    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:24:17.004739 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:24:17.004694    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:24:17.005121 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:24:17.004931    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:24:30.004951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:24:30.004914    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:24:30.005344 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:24:30.005181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:24:43.005490 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:24:43.005437    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:24:43.006954 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:24:43.005719    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:24:58.012027 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:24:58.011989    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:24:58.012522 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:24:58.012225    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:25:13.005240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:13.005210    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:25:13.005716 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:13.005576    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:25:14.129867 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:14.129778    2566 generic.go:356] "Generic (PLEG): container finished" podID="b87cb34a-a29a-467b-af80-46275ddbad60" containerID="2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929" exitCode=1
Jul 16 22:25:14.129867 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:14.129825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929"}
Jul 16 22:25:14.130313 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:14.129872    2566 scope.go:117] "RemoveContainer" containerID="2ca38e5e6d112cd692d8821bf3cd977a46ffbd4f32da026a5b936bd129e28a6e"
Jul 16 22:25:14.130313 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:14.130207    2566 scope.go:117] "RemoveContainer" containerID="2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929"
Jul 16 22:25:14.130521 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:14.130498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:25:14.650986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:14.650953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:25:15.137413 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:15.137387    2566 scope.go:117] "RemoveContainer" containerID="2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929"
Jul 16 22:25:15.137805 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:15.137599    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:25:21.150631 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:21.150590    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:25:21.151105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:21.151086    2566 scope.go:117] "RemoveContainer" containerID="2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929"
Jul 16 22:25:21.151343 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:21.151316    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv_odh-ai-gateway-infra(b87cb34a-a29a-467b-af80-46275ddbad60)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" podUID="b87cb34a-a29a-467b-af80-46275ddbad60"
Jul 16 22:25:25.085266 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:25.085229    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:25:25.085785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:25.085516    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" containerID="cri-o://420edee084d8374dc6f5ae6850ae454f541c7cacb48ff8abdbba17bcbe63d47d" gracePeriod=30
Jul 16 22:25:28.965878 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:28.965831    2566 patch_prober.go:28] interesting pod/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:25:28.966248 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:28.965909    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:25:30.224775 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.224739    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerID="420edee084d8374dc6f5ae6850ae454f541c7cacb48ff8abdbba17bcbe63d47d" exitCode=0
Jul 16 22:25:30.225131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.224789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerDied","Data":"420edee084d8374dc6f5ae6850ae454f541c7cacb48ff8abdbba17bcbe63d47d"}
Jul 16 22:25:30.225131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.224854    2566 scope.go:117] "RemoveContainer" containerID="7783b82565ade2c12feda8c6b0aae6e70bb72f9d33fd696776302beae9e1cbfe"
Jul 16 22:25:30.353285 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.353224    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:25:30.450513 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450673 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450539    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwhkn\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450673 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450673 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450673 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450883 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450883 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450758    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450883 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.450883 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.450829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy\") pod \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\" (UID: \"a6f88309-588c-46ae-bfdb-0ee9f9ec8435\") "
Jul 16 22:25:30.451155 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:25:30.451354 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451333    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.451599 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:25:30.451599 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:25:30.451758 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451618    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:25:30.451885 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.451855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:30.452978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.452956    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:25:30.453084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.453063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:25:30.453084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.453067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:25:30.453197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.453175    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn" (OuterVolumeSpecName: "kube-api-access-wwhkn") pod "a6f88309-588c-46ae-bfdb-0ee9f9ec8435" (UID: "a6f88309-588c-46ae-bfdb-0ee9f9ec8435"). InnerVolumeSpecName "kube-api-access-wwhkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:25:30.552228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552194    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552222    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552232    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552239    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552249    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552257    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552266    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhkn\" (UniqueName: \"kubernetes.io/projected/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-kube-api-access-wwhkn\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:30.552478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:30.552274    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f88309-588c-46ae-bfdb-0ee9f9ec8435-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:31.231328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.231290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6" event={"ID":"a6f88309-588c-46ae-bfdb-0ee9f9ec8435","Type":"ContainerDied","Data":"85ef50166030a8246406f8fb5b7b7686e0d084a25e207b308d78379741230dd1"}
Jul 16 22:25:31.231328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.231325    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"
Jul 16 22:25:31.231825 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.231361    2566 scope.go:117] "RemoveContainer" containerID="420edee084d8374dc6f5ae6850ae454f541c7cacb48ff8abdbba17bcbe63d47d"
Jul 16 22:25:31.260028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.260003    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:25:31.267962 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.267935    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f2f32f5-openshift-default-5f8978fd7-q8bq6"]
Jul 16 22:25:31.609913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.609878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:25:31.754718 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.754687    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:25:31.865132 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.865044    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2glv\" (UniqueName: \"kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv\") pod \"b87cb34a-a29a-467b-af80-46275ddbad60\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") "
Jul 16 22:25:31.865132 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.865089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") pod \"b87cb34a-a29a-467b-af80-46275ddbad60\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") "
Jul 16 22:25:31.865132 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.865130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca\") pod \"b87cb34a-a29a-467b-af80-46275ddbad60\" (UID: \"b87cb34a-a29a-467b-af80-46275ddbad60\") "
Jul 16 22:25:31.865590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.865566    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b87cb34a-a29a-467b-af80-46275ddbad60" (UID: "b87cb34a-a29a-467b-af80-46275ddbad60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:31.867184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.867160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv" (OuterVolumeSpecName: "kube-api-access-w2glv") pod "b87cb34a-a29a-467b-af80-46275ddbad60" (UID: "b87cb34a-a29a-467b-af80-46275ddbad60"). InnerVolumeSpecName "kube-api-access-w2glv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:25:31.867289 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.867245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b87cb34a-a29a-467b-af80-46275ddbad60" (UID: "b87cb34a-a29a-467b-af80-46275ddbad60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:25:31.966242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.966207    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2glv\" (UniqueName: \"kubernetes.io/projected/b87cb34a-a29a-467b-af80-46275ddbad60-kube-api-access-w2glv\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:31.966242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.966236    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87cb34a-a29a-467b-af80-46275ddbad60-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:31.966242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:31.966246    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87cb34a-a29a-467b-af80-46275ddbad60-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:32.016369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.016335    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" path="/var/lib/kubelet/pods/a6f88309-588c-46ae-bfdb-0ee9f9ec8435/volumes"
Jul 16 22:25:32.238186 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.238111    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"
Jul 16 22:25:32.238186 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.238120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv" event={"ID":"b87cb34a-a29a-467b-af80-46275ddbad60","Type":"ContainerDied","Data":"54fe255333688d8c6b2ca06dffab56f54b56b4582eeb6d97572c16bc75a11ba0"}
Jul 16 22:25:32.238697 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.238186    2566 scope.go:117] "RemoveContainer" containerID="2d67624383ce9eaa7b81195bbdcf2f12db0f3a5df4c007e608c2ae9d836a3929"
Jul 16 22:25:32.276134 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.276099    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:25:32.289603 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.289579    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f2f32f5-6fc88bf6d9-4mvmv"]
Jul 16 22:25:32.409950 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.409778    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:25:32.410856 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.410856 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410858    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410883    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410897    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410923    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410931    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410949    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410957    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411021 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.410978    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411142    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411151    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411162    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411170    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f88309-588c-46ae-bfdb-0ee9f9ec8435" containerName="istio-proxy"
Jul 16 22:25:32.411329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.411183    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:32.415010 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.414988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.418818 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.418800    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-67907cc2-openshift-default-dockercfg-xvzlg\""
Jul 16 22:25:32.442479 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.442427    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:25:32.470567 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwzd\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470930 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470930 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470930 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.470930 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.470910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwzd\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.571947 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572267 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.571971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572267 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572267 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572267 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572558 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572558 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.572688 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.572669    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.574271 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.574247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.574377 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.574299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.582003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.581980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.582910 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.582888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwzd\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd\") pod \"e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") " pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.728090 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.728058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:32.879028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.878862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:25:32.881846 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:25:32.881806    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535b0b79_32b8_463f_b4cd_a6ceb686b1a6.slice/crio-cc1127e71cf9af0ce7819d2b3b0a921ae8a28dc13c69381f4a301fbe69c93437 WatchSource:0}: Error finding container cc1127e71cf9af0ce7819d2b3b0a921ae8a28dc13c69381f4a301fbe69c93437: Status 404 returned error can't find the container with id cc1127e71cf9af0ce7819d2b3b0a921ae8a28dc13c69381f4a301fbe69c93437
Jul 16 22:25:32.883942 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.883904    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:32.884048 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.883974    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:32.884048 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:32.884009    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:33.246750 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.246705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerStarted","Data":"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"}
Jul 16 22:25:33.247203 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.246775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerStarted","Data":"cc1127e71cf9af0ce7819d2b3b0a921ae8a28dc13c69381f4a301fbe69c93437"}
Jul 16 22:25:33.273359 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.273307    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podStartSLOduration=1.273293285 podStartE2EDuration="1.273293285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:33.271405394 +0000 UTC m=+2291.781272476" watchObservedRunningTime="2026-07-16 22:25:33.273293285 +0000 UTC m=+2291.783160367"
Jul 16 22:25:33.701562 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.701485    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:25:33.701964 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.701951    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702011 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.701966    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702011 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.701991    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702011 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.702010    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702108 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.702026    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702108 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.702032    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702170 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.702114    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.702170 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.702128    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:33.705018 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.705003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.707973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.707952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:25:33.708099 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.707972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-67907cc2-serving-cert\""
Jul 16 22:25:33.719665 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.719639    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:25:33.729004 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.728979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:33.734098 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.734077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:33.784244 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.784217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckn94\" (UniqueName: \"kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.784387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.784253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.784387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.784284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.885587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.885553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckn94\" (UniqueName: \"kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.885766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.885594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.885766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.885657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.885766 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:33.885689    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-serving-cert: secret "maas-api-e2e-derive-67907cc2-serving-cert" not found
Jul 16 22:25:33.885766 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:33.885740    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls podName:f73b32e1-d664-4d49-9c9c-f5c9215038a5 nodeName:}" failed. No retries permitted until 2026-07-16 22:25:34.385722688 +0000 UTC m=+2292.895589748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls") pod "maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" (UID: "f73b32e1-d664-4d49-9c9c-f5c9215038a5") : secret "maas-api-e2e-derive-67907cc2-serving-cert" not found
Jul 16 22:25:33.886380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.886362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:33.916340 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:33.916322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckn94\" (UniqueName: \"kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:34.010750 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.010719    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" path="/var/lib/kubelet/pods/b87cb34a-a29a-467b-af80-46275ddbad60/volumes"
Jul 16 22:25:34.253201 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.253161    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:34.254088 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.254067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:34.390204 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.390173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:34.393130 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.393101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") pod \"maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:34.616512 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.616450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:34.967921 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:34.967892    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:25:34.969818 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:25:34.969785    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73b32e1_d664_4d49_9c9c_f5c9215038a5.slice/crio-1ab75be3a7f694e1a20f8b7c9d4b501e9b5f7f4fe292c78c77f288c6963602ec WatchSource:0}: Error finding container 1ab75be3a7f694e1a20f8b7c9d4b501e9b5f7f4fe292c78c77f288c6963602ec: Status 404 returned error can't find the container with id 1ab75be3a7f694e1a20f8b7c9d4b501e9b5f7f4fe292c78c77f288c6963602ec
Jul 16 22:25:35.260123 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:35.260091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerStarted","Data":"1ab75be3a7f694e1a20f8b7c9d4b501e9b5f7f4fe292c78c77f288c6963602ec"}
Jul 16 22:25:36.267142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:36.267103    2566 generic.go:356] "Generic (PLEG): container finished" podID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerID="96daeb46421ec62be0d8e75a02d1111c6fd030c84c9eed1b3302b150d64c7e26" exitCode=1
Jul 16 22:25:36.267558 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:36.267192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"96daeb46421ec62be0d8e75a02d1111c6fd030c84c9eed1b3302b150d64c7e26"}
Jul 16 22:25:36.267558 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:36.267542    2566 scope.go:117] "RemoveContainer" containerID="96daeb46421ec62be0d8e75a02d1111c6fd030c84c9eed1b3302b150d64c7e26"
Jul 16 22:25:37.898225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:37.898184    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:37.898931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:37.898909    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87cb34a-a29a-467b-af80-46275ddbad60" containerName="maas-api"
Jul 16 22:25:37.901442 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:37.901419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:37.904314 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:37.904260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:25:37.911700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:37.911676    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:38.030355 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.030320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hltws\" (UID: \"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:38.131379 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.131347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hltws\" (UID: \"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:38.131995 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.131975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-hltws\" (UID: \"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:38.213601 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.213525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:38.283327 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.283291    2566 generic.go:356] "Generic (PLEG): container finished" podID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5" exitCode=1
Jul 16 22:25:38.283492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.283376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"}
Jul 16 22:25:38.283492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.283435    2566 scope.go:117] "RemoveContainer" containerID="96daeb46421ec62be0d8e75a02d1111c6fd030c84c9eed1b3302b150d64c7e26"
Jul 16 22:25:38.283782 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.283763    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:25:38.284159 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:38.284133    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:25:38.357533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:38.357506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:38.358733 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:25:38.358705    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c762a5_8dc4_4dbd_8359_7c36e8973ddb.slice/crio-a2a28273b8cd9c4abe4c618d1f1594acd7ea67069cf2a35cea54f278987ac2f2 WatchSource:0}: Error finding container a2a28273b8cd9c4abe4c618d1f1594acd7ea67069cf2a35cea54f278987ac2f2: Status 404 returned error can't find the container with id a2a28273b8cd9c4abe4c618d1f1594acd7ea67069cf2a35cea54f278987ac2f2
Jul 16 22:25:39.065484 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:39.065430    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c762a5_8dc4_4dbd_8359_7c36e8973ddb.slice/crio-conmon-9af0123b9101ba1e17c120ac9ff106e4a20a4e033b6344ec20e25f330ec150b8.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:25:39.290013 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.289982    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerID="9af0123b9101ba1e17c120ac9ff106e4a20a4e033b6344ec20e25f330ec150b8" exitCode=7
Jul 16 22:25:39.290173 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.290065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerDied","Data":"9af0123b9101ba1e17c120ac9ff106e4a20a4e033b6344ec20e25f330ec150b8"}
Jul 16 22:25:39.290173 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.290127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerStarted","Data":"a2a28273b8cd9c4abe4c618d1f1594acd7ea67069cf2a35cea54f278987ac2f2"}
Jul 16 22:25:39.290282 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.290216    2566 scope.go:117] "RemoveContainer" containerID="9af0123b9101ba1e17c120ac9ff106e4a20a4e033b6344ec20e25f330ec150b8"
Jul 16 22:25:39.291785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.291766    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:25:39.292007 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:39.291981    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:25:39.617597 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:39.617558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:40.306787 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:40.306746    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerID="3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa" exitCode=7
Jul 16 22:25:40.307253 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:40.306828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerDied","Data":"3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa"}
Jul 16 22:25:40.307253 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:40.306922    2566 scope.go:117] "RemoveContainer" containerID="9af0123b9101ba1e17c120ac9ff106e4a20a4e033b6344ec20e25f330ec150b8"
Jul 16 22:25:40.307253 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:40.307085    2566 scope.go:117] "RemoveContainer" containerID="3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa"
Jul 16 22:25:40.307391 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:40.307288    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:25:40.307391 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:40.307346    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hltws_odh-ai-gateway-infra(d5c762a5-8dc4-4dbd-8359-7c36e8973ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb"
Jul 16 22:25:40.307544 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:40.307524    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:25:41.315519 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:41.315487    2566 scope.go:117] "RemoveContainer" containerID="3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa"
Jul 16 22:25:41.315884 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:41.315749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hltws_odh-ai-gateway-infra(d5c762a5-8dc4-4dbd-8359-7c36e8973ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb"
Jul 16 22:25:44.617523 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:44.617490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:25:44.617969 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:44.617861    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:25:44.618059 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:44.618039    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:25:50.294679 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:50.294636    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 22:25:50.295077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:50.294746    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:25:51.388638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.388602    2566 generic.go:356] "Generic (PLEG): container finished" podID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerID="7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795" exitCode=0
Jul 16 22:25:51.389069 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.388677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerDied","Data":"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"}
Jul 16 22:25:51.389125 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.389098    2566 scope.go:117] "RemoveContainer" containerID="7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"
Jul 16 22:25:51.390199 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.390171    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:51.390276 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.390238    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:51.390321 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:51.390288    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:25:52.395987 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:52.395948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerStarted","Data":"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"}
Jul 16 22:25:52.729388 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:52.729279    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:52.729639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:52.729606    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 22:25:52.729752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:52.729676    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 22:25:53.005028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:53.004999    2566 scope.go:117] "RemoveContainer" containerID="3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa"
Jul 16 22:25:53.403888 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:53.403849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerStarted","Data":"7d6fdc121224b621055ab113283f227ce9869fcfec90c0eb47c0f0aabb06287b"}
Jul 16 22:25:53.420926 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:53.420869    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" podStartSLOduration=16.420851687 podStartE2EDuration="16.420851687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:53.419548797 +0000 UTC m=+2311.929415881" watchObservedRunningTime="2026-07-16 22:25:53.420851687 +0000 UTC m=+2311.930718768"
Jul 16 22:25:53.729638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:53.729530    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 22:25:53.729638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:53.729605    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 22:25:54.411369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.411336    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerID="7d6fdc121224b621055ab113283f227ce9869fcfec90c0eb47c0f0aabb06287b" exitCode=7
Jul 16 22:25:54.411798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.411386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerDied","Data":"7d6fdc121224b621055ab113283f227ce9869fcfec90c0eb47c0f0aabb06287b"}
Jul 16 22:25:54.411798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.411432    2566 scope.go:117] "RemoveContainer" containerID="3f1927a1164cdbe6bc4a09e36639629e593b2ee9b4288ea559e3c57f6c1d3dfa"
Jul 16 22:25:54.411798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.411765    2566 scope.go:117] "RemoveContainer" containerID="7d6fdc121224b621055ab113283f227ce9869fcfec90c0eb47c0f0aabb06287b"
Jul 16 22:25:54.412028 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:25:54.412008    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-hltws_odh-ai-gateway-infra(d5c762a5-8dc4-4dbd-8359-7c36e8973ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb"
Jul 16 22:25:54.435975 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.435939    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:54.728798 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.728693    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 22:25:54.729083 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:54.729049    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 22:25:55.604564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.604535    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:55.614577 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.614556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca\") pod \"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb\" (UID: \"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb\") "
Jul 16 22:25:55.614882 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.614857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" (UID: "d5c762a5-8dc4-4dbd-8359-7c36e8973ddb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:55.716177 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.716090    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:25:55.737480 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.731070    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 22:25:55.737480 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:55.731148    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 22:25:56.425048 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.424957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"
Jul 16 22:25:56.425048 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.424991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws" event={"ID":"d5c762a5-8dc4-4dbd-8359-7c36e8973ddb","Type":"ContainerDied","Data":"a2a28273b8cd9c4abe4c618d1f1594acd7ea67069cf2a35cea54f278987ac2f2"}
Jul 16 22:25:56.425280 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.425056    2566 scope.go:117] "RemoveContainer" containerID="7d6fdc121224b621055ab113283f227ce9869fcfec90c0eb47c0f0aabb06287b"
Jul 16 22:25:56.442855 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.442832    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:56.448357 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.448333    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-hltws"]
Jul 16 22:25:56.731841 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.731759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:56.732284 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.732146    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:56.732760 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:56.732741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:25:58.011731 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:25:58.011698    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" path="/var/lib/kubelet/pods/d5c762a5-8dc4-4dbd-8359-7c36e8973ddb/volumes"
Jul 16 22:26:00.005358 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:00.005320    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:26:01.455294 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:01.455262    2566 generic.go:356] "Generic (PLEG): container finished" podID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683" exitCode=1
Jul 16 22:26:01.455679 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:01.455293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"}
Jul 16 22:26:01.455679 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:01.455382    2566 scope.go:117] "RemoveContainer" containerID="69bd1417f636061a77c6240b7566b1e4d50cb39b75bf287048b53037f9af1fc5"
Jul 16 22:26:01.455801 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:01.455769    2566 scope.go:117] "RemoveContainer" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"
Jul 16 22:26:01.456018 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:01.455997    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:04.616931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:04.616892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:26:04.617365 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:04.616962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:26:04.617484 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:04.617405    2566 scope.go:117] "RemoveContainer" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"
Jul 16 22:26:04.617674 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:04.617650    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:19.005102 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:19.005022    2566 scope.go:117] "RemoveContainer" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"
Jul 16 22:26:19.005616 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:19.005215    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:30.005583 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:30.005545    2566 scope.go:117] "RemoveContainer" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"
Jul 16 22:26:30.624162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:30.624124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerStarted","Data":"4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"}
Jul 16 22:26:30.624361 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:30.624342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:26:30.672380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:30.672274    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podStartSLOduration=55.176027446 podStartE2EDuration="57.672258338s" totalImagesPullingTime="2.496230892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:33 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:30.66082879 +0000 UTC m=+2349.170695872" watchObservedRunningTime="2026-07-16 22:26:30.672258338 +0000 UTC m=+2349.182125458"
Jul 16 22:26:31.632011 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:31.631972    2566 generic.go:356] "Generic (PLEG): container finished" podID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057" exitCode=1
Jul 16 22:26:31.632519 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:31.632029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"}
Jul 16 22:26:31.632519 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:31.632109    2566 scope.go:117] "RemoveContainer" containerID="7ac196272ecde8d15b97fa6478ca799637ea1a6ba081fdd85396a19544eb2683"
Jul 16 22:26:31.632519 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:31.632377    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:26:31.632700 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:31.632664    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:32.639146 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:32.639114    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:26:32.639625 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:32.639366    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:34.617540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:34.617503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:26:34.617929 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:34.617908    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:26:34.618120 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:34.618101    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:26:47.005136 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:26:47.005104    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:26:47.005540 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:26:47.005283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:01.005272 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:01.005243    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:27:01.005733 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:01.005415    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:15.004575 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:15.004540    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:27:15.877065 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:15.877030    2566 generic.go:356] "Generic (PLEG): container finished" podID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189" exitCode=1
Jul 16 22:27:15.877207 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:15.877101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"}
Jul 16 22:27:15.877207 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:15.877190    2566 scope.go:117] "RemoveContainer" containerID="4d5fd1bcec4cfeb5a1cdbbeae3d15b05962039abc11b218d19c94fb7aa514057"
Jul 16 22:27:15.877560 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:15.877545    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:27:15.877804 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:15.877782    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:20.624762 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:20.624730    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:27:20.625167 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:20.625089    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:27:20.625344 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:20.625320    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:22.140107 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:22.140077    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:27:22.146111 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:22.146084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:27:24.616710 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:24.616678    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:27:24.617064 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:24.617030    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:27:24.617229 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:24.617210    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:36.005226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.005193    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:27:36.005698 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:36.005404    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:36.711094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711056    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:36.711590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711575    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.711647 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711592    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.711647 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711606    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.711647 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711611    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.711739 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711683    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.711739 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.711693    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:27:36.714376 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.714357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:36.717110 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.717090    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:27:36.724181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.724152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:36.761159 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.761130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4\" (UID: \"1c18a437-f42a-49b1-87bf-43487b2fc5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:36.861959 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.861913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4\" (UID: \"1c18a437-f42a-49b1-87bf-43487b2fc5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:36.862702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:36.862682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4\" (UID: \"1c18a437-f42a-49b1-87bf-43487b2fc5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:37.027577 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:37.027545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:37.163177 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:37.163144    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:37.164604 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:27:37.164577    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c18a437_f42a_49b1_87bf_43487b2fc5f7.slice/crio-ddaaa958301d35546385826efa3699c06c11905009f15dd2e7c7b7b77e231357 WatchSource:0}: Error finding container ddaaa958301d35546385826efa3699c06c11905009f15dd2e7c7b7b77e231357: Status 404 returned error can't find the container with id ddaaa958301d35546385826efa3699c06c11905009f15dd2e7c7b7b77e231357
Jul 16 22:27:38.006402 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:38.006359    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerID="d5e1350bc2aa404c1621f86cc1e698191835e13bb6f47435cebf9b6394e71abf" exitCode=6
Jul 16 22:27:38.006746 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:38.006725    2566 scope.go:117] "RemoveContainer" containerID="d5e1350bc2aa404c1621f86cc1e698191835e13bb6f47435cebf9b6394e71abf"
Jul 16 22:27:38.012181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:38.012157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" event={"ID":"1c18a437-f42a-49b1-87bf-43487b2fc5f7","Type":"ContainerDied","Data":"d5e1350bc2aa404c1621f86cc1e698191835e13bb6f47435cebf9b6394e71abf"}
Jul 16 22:27:38.012282 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:38.012202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" event={"ID":"1c18a437-f42a-49b1-87bf-43487b2fc5f7","Type":"ContainerStarted","Data":"ddaaa958301d35546385826efa3699c06c11905009f15dd2e7c7b7b77e231357"}
Jul 16 22:27:39.013685 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:39.013651    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerID="21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c" exitCode=6
Jul 16 22:27:39.014151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:39.013741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" event={"ID":"1c18a437-f42a-49b1-87bf-43487b2fc5f7","Type":"ContainerDied","Data":"21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c"}
Jul 16 22:27:39.014151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:39.013816    2566 scope.go:117] "RemoveContainer" containerID="d5e1350bc2aa404c1621f86cc1e698191835e13bb6f47435cebf9b6394e71abf"
Jul 16 22:27:39.014151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:39.013916    2566 scope.go:117] "RemoveContainer" containerID="21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c"
Jul 16 22:27:39.014265 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:39.014172    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4_odh-ai-gateway-infra(1c18a437-f42a-49b1-87bf-43487b2fc5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7"
Jul 16 22:27:40.020112 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:40.020079    2566 scope.go:117] "RemoveContainer" containerID="21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c"
Jul 16 22:27:40.020501 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:40.020340    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4_odh-ai-gateway-infra(1c18a437-f42a-49b1-87bf-43487b2fc5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7"
Jul 16 22:27:51.005275 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:51.005242    2566 scope.go:117] "RemoveContainer" containerID="21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c"
Jul 16 22:27:51.005736 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:51.005347    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:27:51.005736 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:51.005566    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:27:52.093492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:52.093442    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerID="9d7cba58cabe93daf667071a8dca6468da1c02b4ee10c0e9aad5f492f1a1c05a" exitCode=6
Jul 16 22:27:52.093951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:52.093518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" event={"ID":"1c18a437-f42a-49b1-87bf-43487b2fc5f7","Type":"ContainerDied","Data":"9d7cba58cabe93daf667071a8dca6468da1c02b4ee10c0e9aad5f492f1a1c05a"}
Jul 16 22:27:52.093951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:52.093590    2566 scope.go:117] "RemoveContainer" containerID="21185ebb8ed453a20a92cc2d409c4dbac432bb735682acd1e4b9eadc60c4577c"
Jul 16 22:27:52.094081 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:52.093965    2566 scope.go:117] "RemoveContainer" containerID="9d7cba58cabe93daf667071a8dca6468da1c02b4ee10c0e9aad5f492f1a1c05a"
Jul 16 22:27:52.094219 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:27:52.094195    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4_odh-ai-gateway-infra(1c18a437-f42a-49b1-87bf-43487b2fc5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7"
Jul 16 22:27:53.142895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:53.142858    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:53.288807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:53.288776    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:53.429591 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:53.429504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca\") pod \"1c18a437-f42a-49b1-87bf-43487b2fc5f7\" (UID: \"1c18a437-f42a-49b1-87bf-43487b2fc5f7\") "
Jul 16 22:27:53.429884 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:53.429860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c18a437-f42a-49b1-87bf-43487b2fc5f7" (UID: "1c18a437-f42a-49b1-87bf-43487b2fc5f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:53.530827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:53.530792    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c18a437-f42a-49b1-87bf-43487b2fc5f7-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:27:54.107967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:54.107933    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"
Jul 16 22:27:54.108138 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:54.107968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4" event={"ID":"1c18a437-f42a-49b1-87bf-43487b2fc5f7","Type":"ContainerDied","Data":"ddaaa958301d35546385826efa3699c06c11905009f15dd2e7c7b7b77e231357"}
Jul 16 22:27:54.108138 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:54.108053    2566 scope.go:117] "RemoveContainer" containerID="9d7cba58cabe93daf667071a8dca6468da1c02b4ee10c0e9aad5f492f1a1c05a"
Jul 16 22:27:54.124917 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:54.124890    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:54.128515 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:54.128493    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-lmlb4"]
Jul 16 22:27:56.011703 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:27:56.011670    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" path="/var/lib/kubelet/pods/1c18a437-f42a-49b1-87bf-43487b2fc5f7/volumes"
Jul 16 22:28:05.004874 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:05.004834    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:28:05.005253 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:28:05.005026    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:28:18.004950 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:18.004906    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:28:18.005422 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:28:18.005128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:28:32.007668 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:32.007633    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:28:32.008121 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:28:32.007841    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm_odh-ai-gateway-infra(f73b32e1-d664-4d49-9c9c-f5c9215038a5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5"
Jul 16 22:28:38.012776 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:38.012743    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:28:38.013166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:38.012944    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" containerID="cri-o://a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a" gracePeriod=30
Jul 16 22:28:41.734582 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:41.734544    2566 patch_prober.go:28] interesting pod/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:28:41.734943 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:41.734604    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:28:43.272326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.272269    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:28:43.324210 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324171    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324281    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324375    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324425    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.324748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:28:43.324748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:28:43.324900 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.324786    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:43.325111 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.325089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.325201 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.325148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwzd\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd\") pod \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\" (UID: \"535b0b79-32b8-463f-b4cd-a6ceb686b1a6\") "
Jul 16 22:28:43.325411 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.325384    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:28:43.325891 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.325851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:28:43.326019 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.325984    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.326019 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326007    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.326153 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326022    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.326153 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326039    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.326153 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326053    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.326944 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:28:43.327061 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:28:43.327061 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.326978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:28:43.328026 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.328002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd" (OuterVolumeSpecName: "kube-api-access-nwwzd") pod "535b0b79-32b8-463f-b4cd-a6ceb686b1a6" (UID: "535b0b79-32b8-463f-b4cd-a6ceb686b1a6"). InnerVolumeSpecName "kube-api-access-nwwzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:28:43.380542 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.380503    2566 generic.go:356] "Generic (PLEG): container finished" podID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerID="a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a" exitCode=0
Jul 16 22:28:43.380675 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.380557    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"
Jul 16 22:28:43.380675 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.380579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerDied","Data":"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"}
Jul 16 22:28:43.380675 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.380635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw" event={"ID":"535b0b79-32b8-463f-b4cd-a6ceb686b1a6","Type":"ContainerDied","Data":"cc1127e71cf9af0ce7819d2b3b0a921ae8a28dc13c69381f4a301fbe69c93437"}
Jul 16 22:28:43.380675 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.380664    2566 scope.go:117] "RemoveContainer" containerID="a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"
Jul 16 22:28:43.390755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.390738    2566 scope.go:117] "RemoveContainer" containerID="7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"
Jul 16 22:28:43.398799 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.398776    2566 scope.go:117] "RemoveContainer" containerID="a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"
Jul 16 22:28:43.399030 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:28:43.399013    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a\": container with ID starting with a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a not found: ID does not exist" containerID="a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"
Jul 16 22:28:43.399074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.399038    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a"} err="failed to get container status \"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a\": rpc error: code = NotFound desc = could not find container \"a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a\": container with ID starting with a8e62a8179f116a5ac7eff69987919ad3b3f154799dc68a5dddf863e737a871a not found: ID does not exist"
Jul 16 22:28:43.399074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.399052    2566 scope.go:117] "RemoveContainer" containerID="7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"
Jul 16 22:28:43.399293 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:28:43.399275    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795\": container with ID starting with 7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795 not found: ID does not exist" containerID="7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"
Jul 16 22:28:43.399352 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.399296    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795"} err="failed to get container status \"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795\": rpc error: code = NotFound desc = could not find container \"7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795\": container with ID starting with 7ceebdf63a7b1a529d48f44b263e8daaf1b0f8b5d99b24e421d60f4d35364795 not found: ID does not exist"
Jul 16 22:28:43.403909 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.403886    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:28:43.406089 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.406066    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-67907cc2-openshift-default-5c7cb568c9-pwxrw"]
Jul 16 22:28:43.427317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.427291    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.427317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.427314    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.427449 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.427323    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:43.427449 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:43.427332    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwzd\" (UniqueName: \"kubernetes.io/projected/535b0b79-32b8-463f-b4cd-a6ceb686b1a6-kube-api-access-nwwzd\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:44.011621 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.011591    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" path="/var/lib/kubelet/pods/535b0b79-32b8-463f-b4cd-a6ceb686b1a6/volumes"
Jul 16 22:28:44.132645 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.132605    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:28:44.276622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.276560    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:28:44.335647 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.335614    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") pod \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") "
Jul 16 22:28:44.335822 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.335671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckn94\" (UniqueName: \"kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94\") pod \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") "
Jul 16 22:28:44.335822 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.335696    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca\") pod \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\" (UID: \"f73b32e1-d664-4d49-9c9c-f5c9215038a5\") "
Jul 16 22:28:44.336113 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.336083    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f73b32e1-d664-4d49-9c9c-f5c9215038a5" (UID: "f73b32e1-d664-4d49-9c9c-f5c9215038a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:44.337959 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.337928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f73b32e1-d664-4d49-9c9c-f5c9215038a5" (UID: "f73b32e1-d664-4d49-9c9c-f5c9215038a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:28:44.337959 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.337928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94" (OuterVolumeSpecName: "kube-api-access-ckn94") pod "f73b32e1-d664-4d49-9c9c-f5c9215038a5" (UID: "f73b32e1-d664-4d49-9c9c-f5c9215038a5"). InnerVolumeSpecName "kube-api-access-ckn94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:28:44.386698 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.386670    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"
Jul 16 22:28:44.386860 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.386717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm" event={"ID":"f73b32e1-d664-4d49-9c9c-f5c9215038a5","Type":"ContainerDied","Data":"1ab75be3a7f694e1a20f8b7c9d4b501e9b5f7f4fe292c78c77f288c6963602ec"}
Jul 16 22:28:44.386860 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.386793    2566 scope.go:117] "RemoveContainer" containerID="ea2ac64647955483a800980ef0f5aaf04ae13041e0d0e3372ae82f4a9c4ae189"
Jul 16 22:28:44.412161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.412134    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:28:44.417441 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.417417    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67907cc2-695f6495c9-hhxmm"]
Jul 16 22:28:44.436657 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.436635    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73b32e1-d664-4d49-9c9c-f5c9215038a5-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:44.436657 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.436658    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckn94\" (UniqueName: \"kubernetes.io/projected/f73b32e1-d664-4d49-9c9c-f5c9215038a5-kube-api-access-ckn94\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:44.436777 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:44.436668    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73b32e1-d664-4d49-9c9c-f5c9215038a5-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:28:46.011313 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:28:46.011274    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" path="/var/lib/kubelet/pods/f73b32e1-d664-4d49-9c9c-f5c9215038a5/volumes"
Jul 16 22:29:05.076571 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.076537    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.076965    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.076976    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.076988    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.076994    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077002    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077009    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077017    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077022    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077028    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077033    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077038    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077043    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077050    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:29:05.077091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077055    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077124    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077131    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077137    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077147    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077156    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c762a5-8dc4-4dbd-8359-7c36e8973ddb" containerName="revoke-keys"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077165    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077170    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077176    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.077843 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.077181    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:05.079388 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.079368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:05.081903 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.081879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:29:05.088247 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.088222    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:05.243006 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.242978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-lrpng\" (UID: \"44aecfca-6bf6-4e94-8abe-d0698f36852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:05.344297 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.344260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-lrpng\" (UID: \"44aecfca-6bf6-4e94-8abe-d0698f36852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:05.344973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.344953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-lrpng\" (UID: \"44aecfca-6bf6-4e94-8abe-d0698f36852a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:05.390930 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.390907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:05.730655 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:05.730630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:05.732908 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:29:05.732874    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44aecfca_6bf6_4e94_8abe_d0698f36852a.slice/crio-287099c4e502807adb032ad2b4f3d166d1af81ae700bcaa6ceb406812e4d481f WatchSource:0}: Error finding container 287099c4e502807adb032ad2b4f3d166d1af81ae700bcaa6ceb406812e4d481f: Status 404 returned error can't find the container with id 287099c4e502807adb032ad2b4f3d166d1af81ae700bcaa6ceb406812e4d481f
Jul 16 22:29:06.513899 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:06.513857    2566 generic.go:356] "Generic (PLEG): container finished" podID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerID="a491cdab64a062b60f8dc6bde0ccdcc20a2ecc47bab675ce95cd5009ef2a3c8d" exitCode=6
Jul 16 22:29:06.514592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:06.514025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerDied","Data":"a491cdab64a062b60f8dc6bde0ccdcc20a2ecc47bab675ce95cd5009ef2a3c8d"}
Jul 16 22:29:06.514592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:06.514120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerStarted","Data":"287099c4e502807adb032ad2b4f3d166d1af81ae700bcaa6ceb406812e4d481f"}
Jul 16 22:29:06.514592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:06.514153    2566 scope.go:117] "RemoveContainer" containerID="a491cdab64a062b60f8dc6bde0ccdcc20a2ecc47bab675ce95cd5009ef2a3c8d"
Jul 16 22:29:07.521221 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:07.521192    2566 generic.go:356] "Generic (PLEG): container finished" podID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerID="20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566" exitCode=6
Jul 16 22:29:07.521633 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:07.521275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerDied","Data":"20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566"}
Jul 16 22:29:07.521633 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:07.521331    2566 scope.go:117] "RemoveContainer" containerID="a491cdab64a062b60f8dc6bde0ccdcc20a2ecc47bab675ce95cd5009ef2a3c8d"
Jul 16 22:29:07.521633 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:07.521586    2566 scope.go:117] "RemoveContainer" containerID="20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566"
Jul 16 22:29:07.521847 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:07.521828    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-lrpng_odh-ai-gateway-infra(44aecfca-6bf6-4e94-8abe-d0698f36852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a"
Jul 16 22:29:08.529055 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:08.529021    2566 scope.go:117] "RemoveContainer" containerID="20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566"
Jul 16 22:29:08.529435 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:08.529234    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-lrpng_odh-ai-gateway-infra(44aecfca-6bf6-4e94-8abe-d0698f36852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a"
Jul 16 22:29:18.109738 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.109659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:18.110117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110105    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:18.110117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110118    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="535b0b79-32b8-463f-b4cd-a6ceb686b1a6" containerName="istio-proxy"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110131    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110137    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110149    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110155    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110161    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:18.110192 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110166    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:18.110385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110233    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73b32e1-d664-4d49-9c9c-f5c9215038a5" containerName="maas-api"
Jul 16 22:29:18.110385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.110241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c18a437-f42a-49b1-87bf-43487b2fc5f7" containerName="revoke-keys"
Jul 16 22:29:18.113171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.113155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:18.122194 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.122160    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:18.265178 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.265133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-57g84\" (UID: \"4f27880c-9e21-47ef-8b2d-d6991bc920e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:18.366409 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.366312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-57g84\" (UID: \"4f27880c-9e21-47ef-8b2d-d6991bc920e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:18.367023 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.366997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-57g84\" (UID: \"4f27880c-9e21-47ef-8b2d-d6991bc920e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:18.424233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.424194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:18.563436 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.563408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:18.564852 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:29:18.564824    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27880c_9e21_47ef_8b2d_d6991bc920e5.slice/crio-fda0c9391746b8202d007e94d24bcd8a97c883de6a1bf02a9387697fd0634e66 WatchSource:0}: Error finding container fda0c9391746b8202d007e94d24bcd8a97c883de6a1bf02a9387697fd0634e66: Status 404 returned error can't find the container with id fda0c9391746b8202d007e94d24bcd8a97c883de6a1bf02a9387697fd0634e66
Jul 16 22:29:18.587611 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:18.587572    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" event={"ID":"4f27880c-9e21-47ef-8b2d-d6991bc920e5","Type":"ContainerStarted","Data":"fda0c9391746b8202d007e94d24bcd8a97c883de6a1bf02a9387697fd0634e66"}
Jul 16 22:29:19.594773 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:19.594737    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerID="bc63494455f94ae4cf045cad3d9cc071c5d813951fce112b4c6c126a363b29b9" exitCode=6
Jul 16 22:29:19.595162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:19.594820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" event={"ID":"4f27880c-9e21-47ef-8b2d-d6991bc920e5","Type":"ContainerDied","Data":"bc63494455f94ae4cf045cad3d9cc071c5d813951fce112b4c6c126a363b29b9"}
Jul 16 22:29:19.595162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:19.595001    2566 scope.go:117] "RemoveContainer" containerID="bc63494455f94ae4cf045cad3d9cc071c5d813951fce112b4c6c126a363b29b9"
Jul 16 22:29:20.005283 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.005247    2566 scope.go:117] "RemoveContainer" containerID="20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566"
Jul 16 22:29:20.602075 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.601984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerStarted","Data":"0eb509f70fc291be2f3b565058bf1fd1b5bcbf4869d9e562a1a9f15d77bce3a9"}
Jul 16 22:29:20.603856 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.603833    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerID="66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5" exitCode=6
Jul 16 22:29:20.603979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.603869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" event={"ID":"4f27880c-9e21-47ef-8b2d-d6991bc920e5","Type":"ContainerDied","Data":"66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5"}
Jul 16 22:29:20.603979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.603904    2566 scope.go:117] "RemoveContainer" containerID="bc63494455f94ae4cf045cad3d9cc071c5d813951fce112b4c6c126a363b29b9"
Jul 16 22:29:20.604068 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.604048    2566 scope.go:117] "RemoveContainer" containerID="66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5"
Jul 16 22:29:20.604259 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:20.604240    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-57g84_odh-ai-gateway-infra(4f27880c-9e21-47ef-8b2d-d6991bc920e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5"
Jul 16 22:29:20.628220 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:20.628174    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" podStartSLOduration=15.628160755 podStartE2EDuration="15.628160755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:29:20.626515893 +0000 UTC m=+2519.136382976" watchObservedRunningTime="2026-07-16 22:29:20.628160755 +0000 UTC m=+2519.138027838"
Jul 16 22:29:21.610865 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.610833    2566 scope.go:117] "RemoveContainer" containerID="66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5"
Jul 16 22:29:21.611303 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:21.611088    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-57g84_odh-ai-gateway-infra(4f27880c-9e21-47ef-8b2d-d6991bc920e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5"
Jul 16 22:29:21.612480 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.612442    2566 generic.go:356] "Generic (PLEG): container finished" podID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerID="0eb509f70fc291be2f3b565058bf1fd1b5bcbf4869d9e562a1a9f15d77bce3a9" exitCode=6
Jul 16 22:29:21.612588 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.612483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerDied","Data":"0eb509f70fc291be2f3b565058bf1fd1b5bcbf4869d9e562a1a9f15d77bce3a9"}
Jul 16 22:29:21.612588 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.612538    2566 scope.go:117] "RemoveContainer" containerID="20d4d0e8f9dfb9f845b9ef8f1ab6455ae5ca6038d1800c39a82005f06c5d8566"
Jul 16 22:29:21.612784 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.612767    2566 scope.go:117] "RemoveContainer" containerID="0eb509f70fc291be2f3b565058bf1fd1b5bcbf4869d9e562a1a9f15d77bce3a9"
Jul 16 22:29:21.612964 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:21.612947    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-lrpng_odh-ai-gateway-infra(44aecfca-6bf6-4e94-8abe-d0698f36852a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a"
Jul 16 22:29:21.641500 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:21.641473    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:22.777542 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:22.777511    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:22.802790 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:22.802764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca\") pod \"44aecfca-6bf6-4e94-8abe-d0698f36852a\" (UID: \"44aecfca-6bf6-4e94-8abe-d0698f36852a\") "
Jul 16 22:29:22.803121 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:22.803101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44aecfca-6bf6-4e94-8abe-d0698f36852a" (UID: "44aecfca-6bf6-4e94-8abe-d0698f36852a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:22.903867 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:22.903782    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44aecfca-6bf6-4e94-8abe-d0698f36852a-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:29:23.626931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:23.626898    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"
Jul 16 22:29:23.627097 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:23.626950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng" event={"ID":"44aecfca-6bf6-4e94-8abe-d0698f36852a","Type":"ContainerDied","Data":"287099c4e502807adb032ad2b4f3d166d1af81ae700bcaa6ceb406812e4d481f"}
Jul 16 22:29:23.627097 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:23.627015    2566 scope.go:117] "RemoveContainer" containerID="0eb509f70fc291be2f3b565058bf1fd1b5bcbf4869d9e562a1a9f15d77bce3a9"
Jul 16 22:29:23.650501 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:23.650478    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:23.657254 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:23.657231    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-lrpng"]
Jul 16 22:29:24.011426 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:24.011391    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" path="/var/lib/kubelet/pods/44aecfca-6bf6-4e94-8abe-d0698f36852a/volumes"
Jul 16 22:29:25.422436 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422397    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:25.422868 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422854    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.422931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422869    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.422931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422892    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.422931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422897    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.423050 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422972    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.423050 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.422980    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:25.426210 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.426195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:25.433550 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.433517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:25.524116 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.524074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx\" (UID: \"d1471fc7-1c5e-42e7-ac3b-af56ac748d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:25.625329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.625296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx\" (UID: \"d1471fc7-1c5e-42e7-ac3b-af56ac748d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:25.625952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.625929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx\" (UID: \"d1471fc7-1c5e-42e7-ac3b-af56ac748d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:25.739373 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:25.739343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:26.082625 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:26.082598    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:26.084774 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:29:26.084744    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1471fc7_1c5e_42e7_ac3b_af56ac748d67.slice/crio-35b07df2a27b6167abbaacf38ae5b1c63245485b7a2f808a471a413b3cd91cea WatchSource:0}: Error finding container 35b07df2a27b6167abbaacf38ae5b1c63245485b7a2f808a471a413b3cd91cea: Status 404 returned error can't find the container with id 35b07df2a27b6167abbaacf38ae5b1c63245485b7a2f808a471a413b3cd91cea
Jul 16 22:29:26.644420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:26.644327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerStarted","Data":"689df9c56b7cddf79f8251e3c1a709b763f5223fb07d4581377807447e875b44"}
Jul 16 22:29:26.644420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:26.644392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerStarted","Data":"35b07df2a27b6167abbaacf38ae5b1c63245485b7a2f808a471a413b3cd91cea"}
Jul 16 22:29:26.662850 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:26.662802    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" podStartSLOduration=1.662786032 podStartE2EDuration="1.662786032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:29:26.662128748 +0000 UTC m=+2525.171995831" watchObservedRunningTime="2026-07-16 22:29:26.662786032 +0000 UTC m=+2525.172653114"
Jul 16 22:29:27.652526 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:27.652494    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerID="689df9c56b7cddf79f8251e3c1a709b763f5223fb07d4581377807447e875b44" exitCode=6
Jul 16 22:29:27.652895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:27.652576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerDied","Data":"689df9c56b7cddf79f8251e3c1a709b763f5223fb07d4581377807447e875b44"}
Jul 16 22:29:27.652895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:27.652757    2566 scope.go:117] "RemoveContainer" containerID="689df9c56b7cddf79f8251e3c1a709b763f5223fb07d4581377807447e875b44"
Jul 16 22:29:28.659789 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:28.659754    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerID="f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073" exitCode=6
Jul 16 22:29:28.660202 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:28.659836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerDied","Data":"f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073"}
Jul 16 22:29:28.660202 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:28.659897    2566 scope.go:117] "RemoveContainer" containerID="689df9c56b7cddf79f8251e3c1a709b763f5223fb07d4581377807447e875b44"
Jul 16 22:29:28.660202 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:28.659974    2566 scope.go:117] "RemoveContainer" containerID="f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073"
Jul 16 22:29:28.660343 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:28.660221    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx_odh-ai-gateway-infra(d1471fc7-1c5e-42e7-ac3b-af56ac748d67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67"
Jul 16 22:29:29.667415 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:29.667389    2566 scope.go:117] "RemoveContainer" containerID="f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073"
Jul 16 22:29:29.667839 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:29.667588    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx_odh-ai-gateway-infra(d1471fc7-1c5e-42e7-ac3b-af56ac748d67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67"
Jul 16 22:29:36.004539 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:36.004502    2566 scope.go:117] "RemoveContainer" containerID="66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5"
Jul 16 22:29:36.713832 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:36.713800    2566 generic.go:356] "Generic (PLEG): container finished" podID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerID="fd3e74970b59f39c6145af10aeca4d406653d8f282c52d39628aa3a5e2ae4a2a" exitCode=6
Jul 16 22:29:36.713961 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:36.713871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" event={"ID":"4f27880c-9e21-47ef-8b2d-d6991bc920e5","Type":"ContainerDied","Data":"fd3e74970b59f39c6145af10aeca4d406653d8f282c52d39628aa3a5e2ae4a2a"}
Jul 16 22:29:36.713961 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:36.713950    2566 scope.go:117] "RemoveContainer" containerID="66eae14e10a3609eec43937d2bbf8aa95178a615636118ef28360edab49395c5"
Jul 16 22:29:36.714197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:36.714180    2566 scope.go:117] "RemoveContainer" containerID="fd3e74970b59f39c6145af10aeca4d406653d8f282c52d39628aa3a5e2ae4a2a"
Jul 16 22:29:36.714488 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:36.714440    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-57g84_odh-ai-gateway-infra(4f27880c-9e21-47ef-8b2d-d6991bc920e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5"
Jul 16 22:29:37.761864 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:37.761833    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:37.904503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:37.904479    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:37.942184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:37.942152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca\") pod \"4f27880c-9e21-47ef-8b2d-d6991bc920e5\" (UID: \"4f27880c-9e21-47ef-8b2d-d6991bc920e5\") "
Jul 16 22:29:37.942548 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:37.942525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f27880c-9e21-47ef-8b2d-d6991bc920e5" (UID: "4f27880c-9e21-47ef-8b2d-d6991bc920e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:38.043230 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.043201    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f27880c-9e21-47ef-8b2d-d6991bc920e5-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:29:38.727985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.727952    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"
Jul 16 22:29:38.728161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.727947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84" event={"ID":"4f27880c-9e21-47ef-8b2d-d6991bc920e5","Type":"ContainerDied","Data":"fda0c9391746b8202d007e94d24bcd8a97c883de6a1bf02a9387697fd0634e66"}
Jul 16 22:29:38.728161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.728097    2566 scope.go:117] "RemoveContainer" containerID="fd3e74970b59f39c6145af10aeca4d406653d8f282c52d39628aa3a5e2ae4a2a"
Jul 16 22:29:38.750007 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.749975    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:38.754640 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:38.754610    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-57g84"]
Jul 16 22:29:40.011005 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:40.010970    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" path="/var/lib/kubelet/pods/4f27880c-9e21-47ef-8b2d-d6991bc920e5/volumes"
Jul 16 22:29:43.005033 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:43.004999    2566 scope.go:117] "RemoveContainer" containerID="f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073"
Jul 16 22:29:43.756492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:43.756442    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerID="130c3cd1ccb1b3464554848c5b43ffc16ab85600116b4d37a9cb25bb64bb953c" exitCode=6
Jul 16 22:29:43.756649 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:43.756490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerDied","Data":"130c3cd1ccb1b3464554848c5b43ffc16ab85600116b4d37a9cb25bb64bb953c"}
Jul 16 22:29:43.756649 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:43.756559    2566 scope.go:117] "RemoveContainer" containerID="f1de689719fa1b6681edc1fc79b5f422ed62a684520397cbde10266c66c1f073"
Jul 16 22:29:43.756865 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:43.756832    2566 scope.go:117] "RemoveContainer" containerID="130c3cd1ccb1b3464554848c5b43ffc16ab85600116b4d37a9cb25bb64bb953c"
Jul 16 22:29:43.757083 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:43.757064    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx_odh-ai-gateway-infra(d1471fc7-1c5e-42e7-ac3b-af56ac748d67)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67"
Jul 16 22:29:44.796537 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:44.796501    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:44.943238 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:44.943215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:45.007748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.007722    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca\") pod \"d1471fc7-1c5e-42e7-ac3b-af56ac748d67\" (UID: \"d1471fc7-1c5e-42e7-ac3b-af56ac748d67\") "
Jul 16 22:29:45.008109 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.008090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1471fc7-1c5e-42e7-ac3b-af56ac748d67" (UID: "d1471fc7-1c5e-42e7-ac3b-af56ac748d67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:45.108784 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.108692    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1471fc7-1c5e-42e7-ac3b-af56ac748d67-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:29:45.770539 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.770500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx" event={"ID":"d1471fc7-1c5e-42e7-ac3b-af56ac748d67","Type":"ContainerDied","Data":"35b07df2a27b6167abbaacf38ae5b1c63245485b7a2f808a471a413b3cd91cea"}
Jul 16 22:29:45.770539 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.770521    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"
Jul 16 22:29:45.770768 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.770562    2566 scope.go:117] "RemoveContainer" containerID="130c3cd1ccb1b3464554848c5b43ffc16ab85600116b4d37a9cb25bb64bb953c"
Jul 16 22:29:45.804510 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.804478    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:45.809245 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:45.809222    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-xd4wx"]
Jul 16 22:29:46.011544 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:46.011512    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" path="/var/lib/kubelet/pods/d1471fc7-1c5e-42e7-ac3b-af56ac748d67/volumes"
Jul 16 22:29:55.167358 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167322    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:29:55.167771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167757    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.167771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167772    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167781    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167786    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167806    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167812    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167819    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167825    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167832    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.167845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167838    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167908    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167916    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167923    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167929    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167937    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44aecfca-6bf6-4e94-8abe-d0698f36852a" containerName="revoke-keys"
Jul 16 22:29:55.168074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.167943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:29:55.171072 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.171055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:29:55.175407 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.175385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:29:55.190725 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.190700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:29:55.302218 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.302183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r\" (UID: \"1f8c4c4f-3a39-4467-89da-79989f48eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:29:55.403723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.403689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r\" (UID: \"1f8c4c4f-3a39-4467-89da-79989f48eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:29:55.404328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.404306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r\" (UID: \"1f8c4c4f-3a39-4467-89da-79989f48eb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:29:55.481377 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.481293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:29:55.841425 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:55.841392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:29:55.845078 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:29:55.845050    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8c4c4f_3a39_4467_89da_79989f48eb20.slice/crio-c4441e4e2fa334aa8eccbbaad2c78046f0d270424d99aa5e3e3c8f9b77ef74c0 WatchSource:0}: Error finding container c4441e4e2fa334aa8eccbbaad2c78046f0d270424d99aa5e3e3c8f9b77ef74c0: Status 404 returned error can't find the container with id c4441e4e2fa334aa8eccbbaad2c78046f0d270424d99aa5e3e3c8f9b77ef74c0
Jul 16 22:29:56.830757 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:56.830725    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerID="bf17d124133e4543d2c88a0198bd2072276297629b3ce3924faccfab7f225ab0" exitCode=6
Jul 16 22:29:56.831329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:56.830796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" event={"ID":"1f8c4c4f-3a39-4467-89da-79989f48eb20","Type":"ContainerDied","Data":"bf17d124133e4543d2c88a0198bd2072276297629b3ce3924faccfab7f225ab0"}
Jul 16 22:29:56.831329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:56.830844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" event={"ID":"1f8c4c4f-3a39-4467-89da-79989f48eb20","Type":"ContainerStarted","Data":"c4441e4e2fa334aa8eccbbaad2c78046f0d270424d99aa5e3e3c8f9b77ef74c0"}
Jul 16 22:29:56.831329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:56.831062    2566 scope.go:117] "RemoveContainer" containerID="bf17d124133e4543d2c88a0198bd2072276297629b3ce3924faccfab7f225ab0"
Jul 16 22:29:57.839594 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:57.839562    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerID="e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab" exitCode=6
Jul 16 22:29:57.840012 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:57.839637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" event={"ID":"1f8c4c4f-3a39-4467-89da-79989f48eb20","Type":"ContainerDied","Data":"e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab"}
Jul 16 22:29:57.840012 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:57.839698    2566 scope.go:117] "RemoveContainer" containerID="bf17d124133e4543d2c88a0198bd2072276297629b3ce3924faccfab7f225ab0"
Jul 16 22:29:57.840012 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:57.839923    2566 scope.go:117] "RemoveContainer" containerID="e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab"
Jul 16 22:29:57.840226 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:57.840200    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r_odh-ai-gateway-infra(1f8c4c4f-3a39-4467-89da-79989f48eb20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20"
Jul 16 22:29:58.846509 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:29:58.846483    2566 scope.go:117] "RemoveContainer" containerID="e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab"
Jul 16 22:29:58.846910 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:29:58.846699    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r_odh-ai-gateway-infra(1f8c4c4f-3a39-4467-89da-79989f48eb20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20"
Jul 16 22:30:00.148177 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148147    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"]
Jul 16 22:30:00.148792 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148772    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:30:00.148890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1471fc7-1c5e-42e7-ac3b-af56ac748d67" containerName="revoke-keys"
Jul 16 22:30:00.148890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:30:00.148890 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148818    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:30:00.149056 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.148966    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27880c-9e21-47ef-8b2d-d6991bc920e5" containerName="revoke-keys"
Jul 16 22:30:00.152084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.152062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"
Jul 16 22:30:00.161497 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.161477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"
Jul 16 22:30:00.168929 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.168905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"]
Jul 16 22:30:00.310879 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.310840    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"]
Jul 16 22:30:00.316732 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:00.316702    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff39cd25_720b_41d0_a8cf_035226b297a5.slice/crio-ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa WatchSource:0}: Error finding container ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa: Status 404 returned error can't find the container with id ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa
Jul 16 22:30:00.857952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.857862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg" event={"ID":"ff39cd25-720b-41d0-a8cf-035226b297a5","Type":"ContainerStarted","Data":"55e342cc617cfb63aa73686902d81f0b3e6a6e9106179bc38a9d9ed60b289610"}
Jul 16 22:30:00.857952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.857928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg" event={"ID":"ff39cd25-720b-41d0-a8cf-035226b297a5","Type":"ContainerStarted","Data":"ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa"}
Jul 16 22:30:00.878983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:00.878932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg" podStartSLOduration=0.87891819 podStartE2EDuration="878.91819ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:00.876507222 +0000 UTC m=+2559.386374303" watchObservedRunningTime="2026-07-16 22:30:00.87891819 +0000 UTC m=+2559.388785272"
Jul 16 22:30:01.864273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:01.864235    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff39cd25-720b-41d0-a8cf-035226b297a5" containerID="55e342cc617cfb63aa73686902d81f0b3e6a6e9106179bc38a9d9ed60b289610" exitCode=0
Jul 16 22:30:01.864707 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:01.864318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg" event={"ID":"ff39cd25-720b-41d0-a8cf-035226b297a5","Type":"ContainerDied","Data":"55e342cc617cfb63aa73686902d81f0b3e6a6e9106179bc38a9d9ed60b289610"}
Jul 16 22:30:03.017182 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:03.017161    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"
Jul 16 22:30:03.876191 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:03.876161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg" event={"ID":"ff39cd25-720b-41d0-a8cf-035226b297a5","Type":"ContainerDied","Data":"ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa"}
Jul 16 22:30:03.876191 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:03.876193    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737350-k2vfg"
Jul 16 22:30:03.876404 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:03.876208    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1916b94d311687ba6de1e525dbddcba50dd24ae22d642a7e7844f3594a7daa"
Jul 16 22:30:11.005180 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:11.005134    2566 scope.go:117] "RemoveContainer" containerID="e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab"
Jul 16 22:30:11.923315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:11.923222    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerID="ef6f79471302f7fc26693ade8ee81674d73d1d1a856016682f5ee7783d2baf35" exitCode=6
Jul 16 22:30:11.923315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:11.923296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" event={"ID":"1f8c4c4f-3a39-4467-89da-79989f48eb20","Type":"ContainerDied","Data":"ef6f79471302f7fc26693ade8ee81674d73d1d1a856016682f5ee7783d2baf35"}
Jul 16 22:30:11.923552 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:11.923357    2566 scope.go:117] "RemoveContainer" containerID="e9418390d1e8543edf8f496984b59c57afbfbc1700c8080dce4df336c8f795ab"
Jul 16 22:30:11.923753 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:11.923723    2566 scope.go:117] "RemoveContainer" containerID="ef6f79471302f7fc26693ade8ee81674d73d1d1a856016682f5ee7783d2baf35"
Jul 16 22:30:11.924021 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:11.923996    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r_odh-ai-gateway-infra(1f8c4c4f-3a39-4467-89da-79989f48eb20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20"
Jul 16 22:30:13.016962 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.016921    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:30:13.159908 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.159881    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:30:13.268170 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.268083    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca\") pod \"1f8c4c4f-3a39-4467-89da-79989f48eb20\" (UID: \"1f8c4c4f-3a39-4467-89da-79989f48eb20\") "
Jul 16 22:30:13.268504 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.268472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f8c4c4f-3a39-4467-89da-79989f48eb20" (UID: "1f8c4c4f-3a39-4467-89da-79989f48eb20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:13.369259 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.369224    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c4c4f-3a39-4467-89da-79989f48eb20-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:13.938499 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.938439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"
Jul 16 22:30:13.938499 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.938444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r" event={"ID":"1f8c4c4f-3a39-4467-89da-79989f48eb20","Type":"ContainerDied","Data":"c4441e4e2fa334aa8eccbbaad2c78046f0d270424d99aa5e3e3c8f9b77ef74c0"}
Jul 16 22:30:13.938761 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.938528    2566 scope.go:117] "RemoveContainer" containerID="ef6f79471302f7fc26693ade8ee81674d73d1d1a856016682f5ee7783d2baf35"
Jul 16 22:30:13.984836 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.984801    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:30:13.990128 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:13.990105    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-7bh4r"]
Jul 16 22:30:14.010778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:14.010753    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" path="/var/lib/kubelet/pods/1f8c4c4f-3a39-4467-89da-79989f48eb20/volumes"
Jul 16 22:30:16.900072 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:16.900481 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900450    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff39cd25-720b-41d0-a8cf-035226b297a5" containerName="cleanup"
Jul 16 22:30:16.900481 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900478    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff39cd25-720b-41d0-a8cf-035226b297a5" containerName="cleanup"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900501    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900506    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900519    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900525    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900536    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900612    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff39cd25-720b-41d0-a8cf-035226b297a5" containerName="cleanup"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900620    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.900658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.900627    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:16.905001 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.904982    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:16.909713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.909688    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:30:16.920895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:16.920873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:17.002525 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.002493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck\" (UID: \"33ccd3a0-e647-4a1b-a41b-5fb13013b2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:17.103587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.103556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck\" (UID: \"33ccd3a0-e647-4a1b-a41b-5fb13013b2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:17.104154 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.104136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck\" (UID: \"33ccd3a0-e647-4a1b-a41b-5fb13013b2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:17.214898 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.214827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:17.377468 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.377431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:17.380379 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:17.380356    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ccd3a0_e647_4a1b_a41b_5fb13013b2df.slice/crio-e51029340b18b68b0d871e865677182c91330a25cb09f97ed674ee30b3d31575 WatchSource:0}: Error finding container e51029340b18b68b0d871e865677182c91330a25cb09f97ed674ee30b3d31575: Status 404 returned error can't find the container with id e51029340b18b68b0d871e865677182c91330a25cb09f97ed674ee30b3d31575
Jul 16 22:30:17.963582 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.963485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerStarted","Data":"f613a9b40926b244ef7fa72cd60690d0bd798c0093303d11fc35c92e333238df"}
Jul 16 22:30:17.963948 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.963586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerStarted","Data":"e51029340b18b68b0d871e865677182c91330a25cb09f97ed674ee30b3d31575"}
Jul 16 22:30:17.988712 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:17.988658    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" podStartSLOduration=1.988642829 podStartE2EDuration="1.988642829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:17.987139556 +0000 UTC m=+2576.497006637" watchObservedRunningTime="2026-07-16 22:30:17.988642829 +0000 UTC m=+2576.498509910"
Jul 16 22:30:18.190495 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.190449    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:18.191103 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.191078    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8c4c4f-3a39-4467-89da-79989f48eb20" containerName="revoke-keys"
Jul 16 22:30:18.194104 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.194087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:18.214295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.214178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:18.214445 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.214385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf\" (UID: \"5f35ea1e-e799-404e-ac41-c2e6acefc680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:18.315529 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.315492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf\" (UID: \"5f35ea1e-e799-404e-ac41-c2e6acefc680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:18.316094 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.316075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf\" (UID: \"5f35ea1e-e799-404e-ac41-c2e6acefc680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:18.504570 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.504538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:18.864549 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.864517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:18.866588 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:18.866561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f35ea1e_e799_404e_ac41_c2e6acefc680.slice/crio-d068dd25d62abdece6093db469abb9fc6bde3da1314c117b3dbfcd2b21b0fe78 WatchSource:0}: Error finding container d068dd25d62abdece6093db469abb9fc6bde3da1314c117b3dbfcd2b21b0fe78: Status 404 returned error can't find the container with id d068dd25d62abdece6093db469abb9fc6bde3da1314c117b3dbfcd2b21b0fe78
Jul 16 22:30:18.972619 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.972581    2566 generic.go:356] "Generic (PLEG): container finished" podID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerID="f613a9b40926b244ef7fa72cd60690d0bd798c0093303d11fc35c92e333238df" exitCode=6
Jul 16 22:30:18.973068 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.972692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerDied","Data":"f613a9b40926b244ef7fa72cd60690d0bd798c0093303d11fc35c92e333238df"}
Jul 16 22:30:18.973068 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.972914    2566 scope.go:117] "RemoveContainer" containerID="f613a9b40926b244ef7fa72cd60690d0bd798c0093303d11fc35c92e333238df"
Jul 16 22:30:18.974409 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:18.974385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" event={"ID":"5f35ea1e-e799-404e-ac41-c2e6acefc680","Type":"ContainerStarted","Data":"d068dd25d62abdece6093db469abb9fc6bde3da1314c117b3dbfcd2b21b0fe78"}
Jul 16 22:30:19.983203 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.983107    2566 generic.go:356] "Generic (PLEG): container finished" podID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerID="73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710" exitCode=6
Jul 16 22:30:19.983642 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.983195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerDied","Data":"73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710"}
Jul 16 22:30:19.983642 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.983274    2566 scope.go:117] "RemoveContainer" containerID="f613a9b40926b244ef7fa72cd60690d0bd798c0093303d11fc35c92e333238df"
Jul 16 22:30:19.983642 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.983514    2566 scope.go:117] "RemoveContainer" containerID="73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710"
Jul 16 22:30:19.983801 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:19.983776    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck_odh-ai-gateway-infra(33ccd3a0-e647-4a1b-a41b-5fb13013b2df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df"
Jul 16 22:30:19.984935 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.984913    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerID="20d05aa26f05ce232e35200b997c03e4a39fc7ab6790994c755665ee9a5b626e" exitCode=6
Jul 16 22:30:19.985029 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.984940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" event={"ID":"5f35ea1e-e799-404e-ac41-c2e6acefc680","Type":"ContainerDied","Data":"20d05aa26f05ce232e35200b997c03e4a39fc7ab6790994c755665ee9a5b626e"}
Jul 16 22:30:19.985183 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:19.985167    2566 scope.go:117] "RemoveContainer" containerID="20d05aa26f05ce232e35200b997c03e4a39fc7ab6790994c755665ee9a5b626e"
Jul 16 22:30:20.991942 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:20.991915    2566 scope.go:117] "RemoveContainer" containerID="73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710"
Jul 16 22:30:20.992347 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:20.992125    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck_odh-ai-gateway-infra(33ccd3a0-e647-4a1b-a41b-5fb13013b2df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df"
Jul 16 22:30:20.993299 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:20.993278    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerID="ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a" exitCode=6
Jul 16 22:30:20.993426 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:20.993361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" event={"ID":"5f35ea1e-e799-404e-ac41-c2e6acefc680","Type":"ContainerDied","Data":"ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a"}
Jul 16 22:30:20.993426 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:20.993417    2566 scope.go:117] "RemoveContainer" containerID="20d05aa26f05ce232e35200b997c03e4a39fc7ab6790994c755665ee9a5b626e"
Jul 16 22:30:20.993614 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:20.993598    2566 scope.go:117] "RemoveContainer" containerID="ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a"
Jul 16 22:30:20.993822 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:20.993800    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf_odh-ai-gateway-infra(5f35ea1e-e799-404e-ac41-c2e6acefc680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680"
Jul 16 22:30:22.000603 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:22.000570    2566 scope.go:117] "RemoveContainer" containerID="ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a"
Jul 16 22:30:22.000977 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:22.000810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf_odh-ai-gateway-infra(5f35ea1e-e799-404e-ac41-c2e6acefc680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680"
Jul 16 22:30:25.522335 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.522288    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:25.527634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.527612    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:25.553576 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.553538    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:25.587151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.587112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5\" (UID: \"93eb4b21-4008-4d3b-9032-3f6913c4aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:25.688611 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.688571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5\" (UID: \"93eb4b21-4008-4d3b-9032-3f6913c4aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:25.689205 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.689185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5\" (UID: \"93eb4b21-4008-4d3b-9032-3f6913c4aeb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:25.838549 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.838428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:25.995040 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:25.995011    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:25.996077 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:25.996048    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93eb4b21_4008_4d3b_9032_3f6913c4aeb7.slice/crio-c2d0fd3ba5692a19fff0fa7ab3c51ecd08d6a2fc42648a1ea9ed960280bb0bc9 WatchSource:0}: Error finding container c2d0fd3ba5692a19fff0fa7ab3c51ecd08d6a2fc42648a1ea9ed960280bb0bc9: Status 404 returned error can't find the container with id c2d0fd3ba5692a19fff0fa7ab3c51ecd08d6a2fc42648a1ea9ed960280bb0bc9
Jul 16 22:30:26.023928 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:26.023889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" event={"ID":"93eb4b21-4008-4d3b-9032-3f6913c4aeb7","Type":"ContainerStarted","Data":"c2d0fd3ba5692a19fff0fa7ab3c51ecd08d6a2fc42648a1ea9ed960280bb0bc9"}
Jul 16 22:30:27.031354 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:27.031317    2566 generic.go:356] "Generic (PLEG): container finished" podID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerID="4fc03e8353d85d0599c954c1b13023fda72fbd90e66f3680fe525f608c0f16fc" exitCode=6
Jul 16 22:30:27.031786 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:27.031409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" event={"ID":"93eb4b21-4008-4d3b-9032-3f6913c4aeb7","Type":"ContainerDied","Data":"4fc03e8353d85d0599c954c1b13023fda72fbd90e66f3680fe525f608c0f16fc"}
Jul 16 22:30:27.031786 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:27.031615    2566 scope.go:117] "RemoveContainer" containerID="4fc03e8353d85d0599c954c1b13023fda72fbd90e66f3680fe525f608c0f16fc"
Jul 16 22:30:28.038141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:28.038105    2566 generic.go:356] "Generic (PLEG): container finished" podID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerID="11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025" exitCode=6
Jul 16 22:30:28.038569 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:28.038148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" event={"ID":"93eb4b21-4008-4d3b-9032-3f6913c4aeb7","Type":"ContainerDied","Data":"11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025"}
Jul 16 22:30:28.038569 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:28.038196    2566 scope.go:117] "RemoveContainer" containerID="4fc03e8353d85d0599c954c1b13023fda72fbd90e66f3680fe525f608c0f16fc"
Jul 16 22:30:28.038569 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:28.038440    2566 scope.go:117] "RemoveContainer" containerID="11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025"
Jul 16 22:30:28.038707 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:28.038669    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5_odh-ai-gateway-infra(93eb4b21-4008-4d3b-9032-3f6913c4aeb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7"
Jul 16 22:30:29.046102 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:29.046076    2566 scope.go:117] "RemoveContainer" containerID="11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025"
Jul 16 22:30:29.046621 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:29.046357    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5_odh-ai-gateway-infra(93eb4b21-4008-4d3b-9032-3f6913c4aeb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7"
Jul 16 22:30:30.619766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.619726    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:30:30.623728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.623701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.635891 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.635864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7ac0bc6b-openshift-default-dockercfg-q4pmg\""
Jul 16 22:30:30.670105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.670063    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:30:30.742772 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.742727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.742956 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.742792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.742956 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.742892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.742956 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.742937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.743161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.742996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.743161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.743057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.743161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.743105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btsd\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.743161 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.743142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.743329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.743172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844279 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844279 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btsd\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844892 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844892 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.844892 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.845058 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.844932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.845058 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.845049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.845134 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.845101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.846852 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.846827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.846985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.846967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.861201 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.861166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btsd\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.864159 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.864131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token\") pod \"e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") " pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:30.935003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:30.934918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:31.197372 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:31.197266    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:30:31.203494 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:31.203449    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38417ec9_2996_4a12_8244_a15672844c2e.slice/crio-0ddb8569f43aa8199791f53a7263f11b5271e20cd7819e8cff4e09b2ebd3b48e WatchSource:0}: Error finding container 0ddb8569f43aa8199791f53a7263f11b5271e20cd7819e8cff4e09b2ebd3b48e: Status 404 returned error can't find the container with id 0ddb8569f43aa8199791f53a7263f11b5271e20cd7819e8cff4e09b2ebd3b48e
Jul 16 22:30:31.205644 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:31.205613    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:31.205717 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:31.205677    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:31.205717 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:31.205711    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:32.066220 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:32.066176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerStarted","Data":"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"}
Jul 16 22:30:32.066748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:32.066247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerStarted","Data":"0ddb8569f43aa8199791f53a7263f11b5271e20cd7819e8cff4e09b2ebd3b48e"}
Jul 16 22:30:32.125322 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:32.125262    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podStartSLOduration=2.1252449589999998 podStartE2EDuration="2.125244959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:32.124529876 +0000 UTC m=+2590.634396958" watchObservedRunningTime="2026-07-16 22:30:32.125244959 +0000 UTC m=+2590.635112040"
Jul 16 22:30:32.935757 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:32.935715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:32.941657 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:32.941618    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:33.072131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:33.072103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:33.073077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:33.073052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:34.004611 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.004579    2566 scope.go:117] "RemoveContainer" containerID="73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710"
Jul 16 22:30:34.004830 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.004675    2566 scope.go:117] "RemoveContainer" containerID="ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a"
Jul 16 22:30:34.437162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.437131    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:30:34.441277 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.441259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.444276 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.444256    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:30:34.444645 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.444630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ac0bc6b-serving-cert\""
Jul 16 22:30:34.475176 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.475139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:30:34.583224 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.583139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.583224 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.583187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.583407 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.583324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gtm\" (UniqueName: \"kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.684790 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.684753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gtm\" (UniqueName: \"kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.684952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.684828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.684952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.684869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.685638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.685608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.687311 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.687288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.700747 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.700719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gtm\" (UniqueName: \"kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm\") pod \"maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:34.751634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:34.751596    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:35.085695 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.085664    2566 generic.go:356] "Generic (PLEG): container finished" podID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerID="04ccc3b502950f46e4b66bd9f71872e331500b7c6284cd92d38df7de39f83fd2" exitCode=6
Jul 16 22:30:35.085860 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.085736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerDied","Data":"04ccc3b502950f46e4b66bd9f71872e331500b7c6284cd92d38df7de39f83fd2"}
Jul 16 22:30:35.085860 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.085804    2566 scope.go:117] "RemoveContainer" containerID="73377041743aa01879a52d84fcdf8ca5882015cf5ab81c5ac55e89f86126d710"
Jul 16 22:30:35.086249 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.086228    2566 scope.go:117] "RemoveContainer" containerID="04ccc3b502950f46e4b66bd9f71872e331500b7c6284cd92d38df7de39f83fd2"
Jul 16 22:30:35.086609 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:35.086573    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck_odh-ai-gateway-infra(33ccd3a0-e647-4a1b-a41b-5fb13013b2df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df"
Jul 16 22:30:35.087971 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.087936    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerID="1221d4e159160ddf032c091453ce9292f04df7dbf8fd2852d394558abfe0bebc" exitCode=6
Jul 16 22:30:35.088053 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.088007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" event={"ID":"5f35ea1e-e799-404e-ac41-c2e6acefc680","Type":"ContainerDied","Data":"1221d4e159160ddf032c091453ce9292f04df7dbf8fd2852d394558abfe0bebc"}
Jul 16 22:30:35.088362 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.088338    2566 scope.go:117] "RemoveContainer" containerID="1221d4e159160ddf032c091453ce9292f04df7dbf8fd2852d394558abfe0bebc"
Jul 16 22:30:35.088627 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:35.088606    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf_odh-ai-gateway-infra(5f35ea1e-e799-404e-ac41-c2e6acefc680)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680"
Jul 16 22:30:35.097782 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.097752    2566 scope.go:117] "RemoveContainer" containerID="ea270d0da87f5790eada4c586dcc8b7f3345032c71988a71d4ab1bc2761f9c8a"
Jul 16 22:30:35.143177 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:30:35.143145    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7524905b_190a_40a9_a5d0_0fa22e5e3281.slice/crio-00b848d13792558aa4bcdcce9cc91438b77e0752a6158655442996e074900b6f WatchSource:0}: Error finding container 00b848d13792558aa4bcdcce9cc91438b77e0752a6158655442996e074900b6f: Status 404 returned error can't find the container with id 00b848d13792558aa4bcdcce9cc91438b77e0752a6158655442996e074900b6f
Jul 16 22:30:35.144722 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.144694    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:30:35.148730 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:35.148709    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:30:36.095593 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.095558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" event={"ID":"7524905b-190a-40a9-a5d0-0fa22e5e3281","Type":"ContainerStarted","Data":"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"}
Jul 16 22:30:36.096050 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.095615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:36.096050 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.095640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" event={"ID":"7524905b-190a-40a9-a5d0-0fa22e5e3281","Type":"ContainerStarted","Data":"00b848d13792558aa4bcdcce9cc91438b77e0752a6158655442996e074900b6f"}
Jul 16 22:30:36.120726 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.120679    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" podStartSLOduration=1.666462294 podStartE2EDuration="2.120661504s" totalImagesPullingTime="454.19921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:36.11906719 +0000 UTC m=+2594.628934276" watchObservedRunningTime="2026-07-16 22:30:36.120661504 +0000 UTC m=+2594.630528587"
Jul 16 22:30:36.161661 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.161623    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:36.200317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.200280    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:36.336824 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.336795    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:36.384228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.384186    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:36.403676 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.403638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca\") pod \"33ccd3a0-e647-4a1b-a41b-5fb13013b2df\" (UID: \"33ccd3a0-e647-4a1b-a41b-5fb13013b2df\") "
Jul 16 22:30:36.404047 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.404023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33ccd3a0-e647-4a1b-a41b-5fb13013b2df" (UID: "33ccd3a0-e647-4a1b-a41b-5fb13013b2df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:36.505069 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.505028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca\") pod \"5f35ea1e-e799-404e-ac41-c2e6acefc680\" (UID: \"5f35ea1e-e799-404e-ac41-c2e6acefc680\") "
Jul 16 22:30:36.505396 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.505375    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ccd3a0-e647-4a1b-a41b-5fb13013b2df-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:36.505547 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.505449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f35ea1e-e799-404e-ac41-c2e6acefc680" (UID: "5f35ea1e-e799-404e-ac41-c2e6acefc680"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:36.606008 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:36.605969    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f35ea1e-e799-404e-ac41-c2e6acefc680-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:37.106028 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.105997    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"
Jul 16 22:30:37.106516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.105992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck" event={"ID":"33ccd3a0-e647-4a1b-a41b-5fb13013b2df","Type":"ContainerDied","Data":"e51029340b18b68b0d871e865677182c91330a25cb09f97ed674ee30b3d31575"}
Jul 16 22:30:37.106516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.106148    2566 scope.go:117] "RemoveContainer" containerID="04ccc3b502950f46e4b66bd9f71872e331500b7c6284cd92d38df7de39f83fd2"
Jul 16 22:30:37.107377 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.107357    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"
Jul 16 22:30:37.107476 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.107358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf" event={"ID":"5f35ea1e-e799-404e-ac41-c2e6acefc680","Type":"ContainerDied","Data":"d068dd25d62abdece6093db469abb9fc6bde3da1314c117b3dbfcd2b21b0fe78"}
Jul 16 22:30:37.115404 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.115385    2566 scope.go:117] "RemoveContainer" containerID="1221d4e159160ddf032c091453ce9292f04df7dbf8fd2852d394558abfe0bebc"
Jul 16 22:30:37.132565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.132534    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:37.137178 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.137147    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-l7cck"]
Jul 16 22:30:37.150863 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.150821    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:37.156780 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:37.156747    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2wfrf"]
Jul 16 22:30:38.014820 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:38.014788    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" path="/var/lib/kubelet/pods/33ccd3a0-e647-4a1b-a41b-5fb13013b2df/volumes"
Jul 16 22:30:38.015163 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:38.015149    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" path="/var/lib/kubelet/pods/5f35ea1e-e799-404e-ac41-c2e6acefc680/volumes"
Jul 16 22:30:42.113232 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:42.113192    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:44.004970 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:44.004937    2566 scope.go:117] "RemoveContainer" containerID="11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025"
Jul 16 22:30:45.160972 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:45.160938    2566 generic.go:356] "Generic (PLEG): container finished" podID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerID="f24a1712fafe37dbebcf11eca064544aa3f9996707ddac7934a9ab37e850068c" exitCode=6
Jul 16 22:30:45.161386 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:45.161011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" event={"ID":"93eb4b21-4008-4d3b-9032-3f6913c4aeb7","Type":"ContainerDied","Data":"f24a1712fafe37dbebcf11eca064544aa3f9996707ddac7934a9ab37e850068c"}
Jul 16 22:30:45.161386 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:45.161071    2566 scope.go:117] "RemoveContainer" containerID="11bdfb90821a2c522f43025efc04c27d7e15313a77401b09cb79c1d642e6f025"
Jul 16 22:30:45.161520 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:45.161508    2566 scope.go:117] "RemoveContainer" containerID="f24a1712fafe37dbebcf11eca064544aa3f9996707ddac7934a9ab37e850068c"
Jul 16 22:30:45.161775 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:45.161753    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5_odh-ai-gateway-infra(93eb4b21-4008-4d3b-9032-3f6913c4aeb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7"
Jul 16 22:30:46.209227 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:46.209189    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:46.407240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:46.407214    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:46.506977 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:46.506949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca\") pod \"93eb4b21-4008-4d3b-9032-3f6913c4aeb7\" (UID: \"93eb4b21-4008-4d3b-9032-3f6913c4aeb7\") "
Jul 16 22:30:46.507272 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:46.507249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93eb4b21-4008-4d3b-9032-3f6913c4aeb7" (UID: "93eb4b21-4008-4d3b-9032-3f6913c4aeb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:46.607788 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:46.607753    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93eb4b21-4008-4d3b-9032-3f6913c4aeb7-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:47.176379 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:47.176318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"
Jul 16 22:30:47.176600 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:47.176329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5" event={"ID":"93eb4b21-4008-4d3b-9032-3f6913c4aeb7","Type":"ContainerDied","Data":"c2d0fd3ba5692a19fff0fa7ab3c51ecd08d6a2fc42648a1ea9ed960280bb0bc9"}
Jul 16 22:30:47.176600 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:47.176542    2566 scope.go:117] "RemoveContainer" containerID="f24a1712fafe37dbebcf11eca064544aa3f9996707ddac7934a9ab37e850068c"
Jul 16 22:30:47.220875 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:47.220840    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:47.236725 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:47.236687    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-q8rv5"]
Jul 16 22:30:48.013700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:48.013664    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" path="/var/lib/kubelet/pods/93eb4b21-4008-4d3b-9032-3f6913c4aeb7/volumes"
Jul 16 22:30:49.114663 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:49.114615    2566 patch_prober.go:28] interesting pod/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 22:30:49.115044 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:49.114703    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:30:50.197707 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.197622    2566 generic.go:356] "Generic (PLEG): container finished" podID="38417ec9-2996-4a12-8244-a15672844c2e" containerID="918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f" exitCode=0
Jul 16 22:30:50.197707 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.197697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerDied","Data":"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"}
Jul 16 22:30:50.198430 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.198163    2566 scope.go:117] "RemoveContainer" containerID="918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"
Jul 16 22:30:50.199195 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.199165    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:50.199319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.199259    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:50.199388 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:50.199347    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:30:51.205123 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:51.205079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerStarted","Data":"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"}
Jul 16 22:30:51.936014 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:51.935960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:51.936389 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:51.936357    2566 patch_prober.go:28] interesting pod/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 16 22:30:51.936667 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:51.936634    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 16 22:30:52.935731 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:52.935690    2566 patch_prober.go:28] interesting pod/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 16 22:30:52.936342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:52.936307    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 16 22:30:53.936427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:53.935960    2566 patch_prober.go:28] interesting pod/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 16 22:30:53.936427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:53.936045    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 16 22:30:54.935869 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:54.935831    2566 patch_prober.go:28] interesting pod/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 16 22:30:54.936178 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:54.936146    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 16 22:30:55.938557 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:55.938516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:55.938967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:55.938808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:55.939466 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:55.939432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:30:57.280612 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.280576    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:30:57.281063 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.280776    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" podUID="7524905b-190a-40a9-a5d0-0fa22e5e3281" containerName="maas-api" containerID="cri-o://7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164" gracePeriod=30
Jul 16 22:30:57.548162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.548135    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:57.614881 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.614840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls\") pod \"7524905b-190a-40a9-a5d0-0fa22e5e3281\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") "
Jul 16 22:30:57.615071 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.614890    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca\") pod \"7524905b-190a-40a9-a5d0-0fa22e5e3281\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") "
Jul 16 22:30:57.615071 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.614930    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9gtm\" (UniqueName: \"kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm\") pod \"7524905b-190a-40a9-a5d0-0fa22e5e3281\" (UID: \"7524905b-190a-40a9-a5d0-0fa22e5e3281\") "
Jul 16 22:30:57.615327 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.615301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7524905b-190a-40a9-a5d0-0fa22e5e3281" (UID: "7524905b-190a-40a9-a5d0-0fa22e5e3281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:57.616979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.616952    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm" (OuterVolumeSpecName: "kube-api-access-k9gtm") pod "7524905b-190a-40a9-a5d0-0fa22e5e3281" (UID: "7524905b-190a-40a9-a5d0-0fa22e5e3281"). InnerVolumeSpecName "kube-api-access-k9gtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:30:57.617081 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.616992    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7524905b-190a-40a9-a5d0-0fa22e5e3281" (UID: "7524905b-190a-40a9-a5d0-0fa22e5e3281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:30:57.716336 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.716301    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7524905b-190a-40a9-a5d0-0fa22e5e3281-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:57.716336 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.716336    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7524905b-190a-40a9-a5d0-0fa22e5e3281-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:57.716592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:57.716345    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9gtm\" (UniqueName: \"kubernetes.io/projected/7524905b-190a-40a9-a5d0-0fa22e5e3281-kube-api-access-k9gtm\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:30:58.246245 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.246206    2566 generic.go:356] "Generic (PLEG): container finished" podID="7524905b-190a-40a9-a5d0-0fa22e5e3281" containerID="7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164" exitCode=0
Jul 16 22:30:58.246245 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.246258    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"
Jul 16 22:30:58.246494 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.246295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" event={"ID":"7524905b-190a-40a9-a5d0-0fa22e5e3281","Type":"ContainerDied","Data":"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"}
Jul 16 22:30:58.246494 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.246359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2" event={"ID":"7524905b-190a-40a9-a5d0-0fa22e5e3281","Type":"ContainerDied","Data":"00b848d13792558aa4bcdcce9cc91438b77e0752a6158655442996e074900b6f"}
Jul 16 22:30:58.246494 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.246385    2566 scope.go:117] "RemoveContainer" containerID="7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"
Jul 16 22:30:58.255889 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.255868    2566 scope.go:117] "RemoveContainer" containerID="7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"
Jul 16 22:30:58.256164 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:30:58.256142    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164\": container with ID starting with 7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164 not found: ID does not exist" containerID="7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"
Jul 16 22:30:58.256224 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.256173    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164"} err="failed to get container status \"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164\": rpc error: code = NotFound desc = could not find container \"7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164\": container with ID starting with 7aabe386c75ebacb7946423bd9d983729b1ea8b23985323b8a9c4dfd16aee164 not found: ID does not exist"
Jul 16 22:30:58.266251 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.266219    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:30:58.274140 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:30:58.274069    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ac0bc6b-6bf5bb6c88-4mjj2"]
Jul 16 22:31:00.011842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:00.011806    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7524905b-190a-40a9-a5d0-0fa22e5e3281" path="/var/lib/kubelet/pods/7524905b-190a-40a9-a5d0-0fa22e5e3281/volumes"
Jul 16 22:31:12.564444 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:12.564407    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:31:12.564888 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:12.564705    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy" containerID="cri-o://f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444" gracePeriod=30
Jul 16 22:31:17.832586 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.832523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:31:17.930338 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930409    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7btsd\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket\") pod \"38417ec9-2996-4a12-8244-a15672844c2e\" (UID: \"38417ec9-2996-4a12-8244-a15672844c2e\") "
Jul 16 22:31:17.930815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:17.931036 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.930931    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:17.931036 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.931021    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:17.931141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.931044    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:17.931141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.931111    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:17.931257 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.931189    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:17.931257 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.931196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:17.932771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.932744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:17.932950 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.932925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd" (OuterVolumeSpecName: "kube-api-access-7btsd") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "kube-api-access-7btsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:17.933117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.933091    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:17.933117 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:17.933095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38417ec9-2996-4a12-8244-a15672844c2e" (UID: "38417ec9-2996-4a12-8244-a15672844c2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:31:18.031785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031752    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031779    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38417ec9-2996-4a12-8244-a15672844c2e-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031788    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031797    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btsd\" (UniqueName: \"kubernetes.io/projected/38417ec9-2996-4a12-8244-a15672844c2e-kube-api-access-7btsd\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031806    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38417ec9-2996-4a12-8244-a15672844c2e-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031814    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.031979 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.031822    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38417ec9-2996-4a12-8244-a15672844c2e-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:18.376145 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.376105    2566 generic.go:356] "Generic (PLEG): container finished" podID="38417ec9-2996-4a12-8244-a15672844c2e" containerID="f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444" exitCode=0
Jul 16 22:31:18.376320 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.376158    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"
Jul 16 22:31:18.376320 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.376194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerDied","Data":"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"}
Jul 16 22:31:18.376320 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.376255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh" event={"ID":"38417ec9-2996-4a12-8244-a15672844c2e","Type":"ContainerDied","Data":"0ddb8569f43aa8199791f53a7263f11b5271e20cd7819e8cff4e09b2ebd3b48e"}
Jul 16 22:31:18.376320 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.376285    2566 scope.go:117] "RemoveContainer" containerID="f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"
Jul 16 22:31:18.386270 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.386237    2566 scope.go:117] "RemoveContainer" containerID="918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"
Jul 16 22:31:18.395715 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.395693    2566 scope.go:117] "RemoveContainer" containerID="f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"
Jul 16 22:31:18.396074 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:18.396050    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444\": container with ID starting with f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444 not found: ID does not exist" containerID="f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"
Jul 16 22:31:18.396139 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.396084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444"} err="failed to get container status \"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444\": rpc error: code = NotFound desc = could not find container \"f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444\": container with ID starting with f0d2befe4f491000ba2deea07e43b866e0bd9cfc815f7ebeba3df9b9dcffd444 not found: ID does not exist"
Jul 16 22:31:18.396139 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.396101    2566 scope.go:117] "RemoveContainer" containerID="918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"
Jul 16 22:31:18.396348 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:18.396328    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f\": container with ID starting with 918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f not found: ID does not exist" containerID="918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"
Jul 16 22:31:18.396397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.396359    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f"} err="failed to get container status \"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f\": rpc error: code = NotFound desc = could not find container \"918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f\": container with ID starting with 918d205645a4807b910b4fd6fd2ecff804353655fd489ffb74de6ce8847f6c8f not found: ID does not exist"
Jul 16 22:31:18.423996 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.423953    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:31:18.431584 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:18.431551    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7ac0bc6b-openshift-default-77956ff459-df4mh"]
Jul 16 22:31:20.011934 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:20.011899    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38417ec9-2996-4a12-8244-a15672844c2e" path="/var/lib/kubelet/pods/38417ec9-2996-4a12-8244-a15672844c2e/volumes"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.606920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.607981    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608004    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608037    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608060    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7524905b-190a-40a9-a5d0-0fa22e5e3281" containerName="maas-api"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608069    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7524905b-190a-40a9-a5d0-0fa22e5e3281" containerName="maas-api"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608097    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608105    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608122    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608156    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608164    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608180    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608189    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608205    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608213    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608223    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608249    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608257    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608286    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608500    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608512    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608530    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608539    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608547    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608556    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608571    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608582    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7524905b-190a-40a9-a5d0-0fa22e5e3281" containerName="maas-api"
Jul 16 22:31:25.610564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.608592    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="93eb4b21-4008-4d3b-9032-3f6913c4aeb7" containerName="revoke-keys"
Jul 16 22:31:25.615574 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.615552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:25.619142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.619120    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:31:25.623927 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.623903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:25.699724 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.699687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r\" (UID: \"dc401023-e241-4435-b359-2591779774ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:25.800417 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.800383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r\" (UID: \"dc401023-e241-4435-b359-2591779774ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:25.800998 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.800977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r\" (UID: \"dc401023-e241-4435-b359-2591779774ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:25.927563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:25.927488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:26.074151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:26.074123    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:26.075672 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:31:26.075649    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc401023_e241_4435_b359_2591779774ff.slice/crio-0c515fd53fb68f6d498adfc5ac91cb0de2ac866fc4db82ffc21043e7fd0fb9dc WatchSource:0}: Error finding container 0c515fd53fb68f6d498adfc5ac91cb0de2ac866fc4db82ffc21043e7fd0fb9dc: Status 404 returned error can't find the container with id 0c515fd53fb68f6d498adfc5ac91cb0de2ac866fc4db82ffc21043e7fd0fb9dc
Jul 16 22:31:26.426826 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:26.426786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerStarted","Data":"baf8286de6c3b767fc26c7c8f8f5286d2ded700c05b47095ac99baa677cd0b20"}
Jul 16 22:31:26.427087 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:26.427060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerStarted","Data":"0c515fd53fb68f6d498adfc5ac91cb0de2ac866fc4db82ffc21043e7fd0fb9dc"}
Jul 16 22:31:26.455624 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:26.455566    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" podStartSLOduration=1.455549956 podStartE2EDuration="1.455549956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:26.452279458 +0000 UTC m=+2644.962146540" watchObservedRunningTime="2026-07-16 22:31:26.455549956 +0000 UTC m=+2644.965417037"
Jul 16 22:31:27.433956 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:27.433917    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc401023-e241-4435-b359-2591779774ff" containerID="baf8286de6c3b767fc26c7c8f8f5286d2ded700c05b47095ac99baa677cd0b20" exitCode=6
Jul 16 22:31:27.434366 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:27.433964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerDied","Data":"baf8286de6c3b767fc26c7c8f8f5286d2ded700c05b47095ac99baa677cd0b20"}
Jul 16 22:31:27.434366 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:27.434136    2566 scope.go:117] "RemoveContainer" containerID="baf8286de6c3b767fc26c7c8f8f5286d2ded700c05b47095ac99baa677cd0b20"
Jul 16 22:31:28.441193 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:28.441104    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc401023-e241-4435-b359-2591779774ff" containerID="43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43" exitCode=6
Jul 16 22:31:28.441193 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:28.441185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerDied","Data":"43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43"}
Jul 16 22:31:28.441639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:28.441236    2566 scope.go:117] "RemoveContainer" containerID="baf8286de6c3b767fc26c7c8f8f5286d2ded700c05b47095ac99baa677cd0b20"
Jul 16 22:31:28.441639 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:28.441473    2566 scope.go:117] "RemoveContainer" containerID="43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43"
Jul 16 22:31:28.441747 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:28.441727    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r_odh-ai-gateway-infra(dc401023-e241-4435-b359-2591779774ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" podUID="dc401023-e241-4435-b359-2591779774ff"
Jul 16 22:31:29.448414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:29.448384    2566 scope.go:117] "RemoveContainer" containerID="43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43"
Jul 16 22:31:29.448881 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:29.448725    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r_odh-ai-gateway-infra(dc401023-e241-4435-b359-2591779774ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" podUID="dc401023-e241-4435-b359-2591779774ff"
Jul 16 22:31:35.312309 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312272    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:35.312755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312740    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:35.312755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312755    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:35.312830 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312824    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ccd3a0-e647-4a1b-a41b-5fb13013b2df" containerName="revoke-keys"
Jul 16 22:31:35.312872 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312833    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f35ea1e-e799-404e-ac41-c2e6acefc680" containerName="revoke-keys"
Jul 16 22:31:35.312872 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.312845    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38417ec9-2996-4a12-8244-a15672844c2e" containerName="istio-proxy"
Jul 16 22:31:35.315987 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.315970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:35.330231 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.330204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:35.375599 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.375564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nn88x\" (UID: \"489a29b3-bfcb-4db5-919e-f7d0f4f50880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:35.476758 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.476716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nn88x\" (UID: \"489a29b3-bfcb-4db5-919e-f7d0f4f50880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:35.477363 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.477341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nn88x\" (UID: \"489a29b3-bfcb-4db5-919e-f7d0f4f50880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:35.626989 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.626902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:35.806678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:35.806652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:36.485952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:36.485849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerStarted","Data":"14f30285505abd956257910d123ce1b05efa790c314fb28125a66db40e36b9a6"}
Jul 16 22:31:36.485952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:36.485908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerStarted","Data":"d4d145dd16822a8152db4edb8316c7cae74b761ba1bd66241c7b46ce665113ed"}
Jul 16 22:31:36.525801 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:36.525743    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" podStartSLOduration=1.525725566 podStartE2EDuration="1.525725566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:36.52448335 +0000 UTC m=+2655.034350434" watchObservedRunningTime="2026-07-16 22:31:36.525725566 +0000 UTC m=+2655.035592652"
Jul 16 22:31:37.492184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.492149    2566 generic.go:356] "Generic (PLEG): container finished" podID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerID="14f30285505abd956257910d123ce1b05efa790c314fb28125a66db40e36b9a6" exitCode=6
Jul 16 22:31:37.492627 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.492245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerDied","Data":"14f30285505abd956257910d123ce1b05efa790c314fb28125a66db40e36b9a6"}
Jul 16 22:31:37.492627 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.492384    2566 scope.go:117] "RemoveContainer" containerID="14f30285505abd956257910d123ce1b05efa790c314fb28125a66db40e36b9a6"
Jul 16 22:31:37.980900 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.980861    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:31:37.984554 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.984528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:37.997700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:37.997677    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-601103b4-openshift-default-dockercfg-8kmgm\""
Jul 16 22:31:38.000083 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000184    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000413 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4czx\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.000413 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.000405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.015037 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.015002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:31:38.100928 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.100832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.100928 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.100887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.100928 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.100917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4czx\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.100939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.100985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101514 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101514 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101514 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101350    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101514 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.101996 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.101949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.103504 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.103478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.104134 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.104112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.117842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.117811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4czx\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.120312 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.120292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token\") pod \"e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") " pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.295191 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.295159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:38.468033 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.468002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:31:38.469847 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:31:38.469821    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b83187_6428_4f07_bf99_265637affd0c.slice/crio-c3b0fc9c1e8e823208e5d3f3819fc9c1f3da0ef227b223b9d99812564a054cbf WatchSource:0}: Error finding container c3b0fc9c1e8e823208e5d3f3819fc9c1f3da0ef227b223b9d99812564a054cbf: Status 404 returned error can't find the container with id c3b0fc9c1e8e823208e5d3f3819fc9c1f3da0ef227b223b9d99812564a054cbf
Jul 16 22:31:38.472007 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.471978    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:38.472098 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.472055    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:38.472156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.472099    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:38.499603 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.499576    2566 generic.go:356] "Generic (PLEG): container finished" podID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerID="c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c" exitCode=6
Jul 16 22:31:38.499968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.499656    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerDied","Data":"c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c"}
Jul 16 22:31:38.499968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.499722    2566 scope.go:117] "RemoveContainer" containerID="14f30285505abd956257910d123ce1b05efa790c314fb28125a66db40e36b9a6"
Jul 16 22:31:38.499968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.499863    2566 scope.go:117] "RemoveContainer" containerID="c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c"
Jul 16 22:31:38.500148 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:38.500104    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nn88x_odh-ai-gateway-infra(489a29b3-bfcb-4db5-919e-f7d0f4f50880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880"
Jul 16 22:31:38.500970 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:38.500953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerStarted","Data":"c3b0fc9c1e8e823208e5d3f3819fc9c1f3da0ef227b223b9d99812564a054cbf"}
Jul 16 22:31:39.508681 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:39.508650    2566 scope.go:117] "RemoveContainer" containerID="c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c"
Jul 16 22:31:39.509123 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:39.508908    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nn88x_odh-ai-gateway-infra(489a29b3-bfcb-4db5-919e-f7d0f4f50880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880"
Jul 16 22:31:39.510035 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:39.510008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerStarted","Data":"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"}
Jul 16 22:31:39.573138 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:39.573086    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podStartSLOduration=2.573070019 podStartE2EDuration="2.573070019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:39.571318666 +0000 UTC m=+2658.081185747" watchObservedRunningTime="2026-07-16 22:31:39.573070019 +0000 UTC m=+2658.082937101"
Jul 16 22:31:40.295857 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:40.295821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:40.300713 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:40.300686    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:40.523179 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:40.523141    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:40.524262 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:40.524239    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:42.093088 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.093053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:31:42.097067 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.097051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.110401 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.110380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:31:42.126411 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.126385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-601103b4-serving-cert\""
Jul 16 22:31:42.139494 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.139436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:31:42.142600 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.142567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.142742 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.142654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.142819 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.142776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvkz7\" (UniqueName: \"kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.243823 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.243786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvkz7\" (UniqueName: \"kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.243990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.243855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.243990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.243893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.244077 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:42.244024    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-serving-cert: secret "maas-api-e2e-mt-601103b4-serving-cert" not found
Jul 16 22:31:42.244077 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:42.244075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls podName:b2212878-597d-49bb-9980-25f7dea0505a nodeName:}" failed. No retries permitted until 2026-07-16 22:31:42.744060245 +0000 UTC m=+2661.253927310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls") pod "maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" (UID: "b2212878-597d-49bb-9980-25f7dea0505a") : secret "maas-api-e2e-mt-601103b4-serving-cert" not found
Jul 16 22:31:42.244565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.244545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.280261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.280227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvkz7\" (UniqueName: \"kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.749294 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.749256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:42.751640 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:42.751621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") pod \"maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:43.005361 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:43.005280    2566 scope.go:117] "RemoveContainer" containerID="43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43"
Jul 16 22:31:43.007759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:43.007737    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:43.190968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:43.190935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:31:43.193040 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:31:43.193005    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2212878_597d_49bb_9980_25f7dea0505a.slice/crio-755c59a2583cedabae5807bb3b8b7376d1932bd3e06dd1888948cf2335df9a17 WatchSource:0}: Error finding container 755c59a2583cedabae5807bb3b8b7376d1932bd3e06dd1888948cf2335df9a17: Status 404 returned error can't find the container with id 755c59a2583cedabae5807bb3b8b7376d1932bd3e06dd1888948cf2335df9a17
Jul 16 22:31:43.541119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:43.541076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerStarted","Data":"e1cdab82da2e3a11ecec99a01e27399c5d9c73926ab1b4f865bacf1789515dd0"}
Jul 16 22:31:43.542350 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:43.542316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" event={"ID":"b2212878-597d-49bb-9980-25f7dea0505a","Type":"ContainerStarted","Data":"755c59a2583cedabae5807bb3b8b7376d1932bd3e06dd1888948cf2335df9a17"}
Jul 16 22:31:44.549070 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.549037    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc401023-e241-4435-b359-2591779774ff" containerID="e1cdab82da2e3a11ecec99a01e27399c5d9c73926ab1b4f865bacf1789515dd0" exitCode=6
Jul 16 22:31:44.549533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.549108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerDied","Data":"e1cdab82da2e3a11ecec99a01e27399c5d9c73926ab1b4f865bacf1789515dd0"}
Jul 16 22:31:44.549533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.549173    2566 scope.go:117] "RemoveContainer" containerID="43c37165d708803d145235d920948684d21d5715a400119d168cd1aa9239ff43"
Jul 16 22:31:44.549533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.549503    2566 scope.go:117] "RemoveContainer" containerID="e1cdab82da2e3a11ecec99a01e27399c5d9c73926ab1b4f865bacf1789515dd0"
Jul 16 22:31:44.549818 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:44.549785    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r_odh-ai-gateway-infra(dc401023-e241-4435-b359-2591779774ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" podUID="dc401023-e241-4435-b359-2591779774ff"
Jul 16 22:31:44.550806 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.550783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" event={"ID":"b2212878-597d-49bb-9980-25f7dea0505a","Type":"ContainerStarted","Data":"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"}
Jul 16 22:31:44.550903 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.550845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:44.591187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.591154    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:44.721352 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:44.721297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" podStartSLOduration=2.337105411 podStartE2EDuration="2.721280915s" totalImagesPullingTime="384.175504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:44.701860363 +0000 UTC m=+2663.211727444" watchObservedRunningTime="2026-07-16 22:31:44.721280915 +0000 UTC m=+2663.231147996"
Jul 16 22:31:45.707907 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:45.707882    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:45.778983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:45.778954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca\") pod \"dc401023-e241-4435-b359-2591779774ff\" (UID: \"dc401023-e241-4435-b359-2591779774ff\") "
Jul 16 22:31:45.779313 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:45.779288    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc401023-e241-4435-b359-2591779774ff" (UID: "dc401023-e241-4435-b359-2591779774ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:45.880156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:45.880074    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc401023-e241-4435-b359-2591779774ff-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:46.565629 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:46.565596    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"
Jul 16 22:31:46.565791 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:46.565602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r" event={"ID":"dc401023-e241-4435-b359-2591779774ff","Type":"ContainerDied","Data":"0c515fd53fb68f6d498adfc5ac91cb0de2ac866fc4db82ffc21043e7fd0fb9dc"}
Jul 16 22:31:46.565791 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:46.565729    2566 scope.go:117] "RemoveContainer" containerID="e1cdab82da2e3a11ecec99a01e27399c5d9c73926ab1b4f865bacf1789515dd0"
Jul 16 22:31:46.596805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:46.596770    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:46.603539 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:46.603505    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-x557r"]
Jul 16 22:31:48.011691 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:48.011656    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc401023-e241-4435-b359-2591779774ff" path="/var/lib/kubelet/pods/dc401023-e241-4435-b359-2591779774ff/volumes"
Jul 16 22:31:50.565014 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:50.564979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:31:54.004525 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:54.004496    2566 scope.go:117] "RemoveContainer" containerID="c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c"
Jul 16 22:31:54.616516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:54.616412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerStarted","Data":"91e76c631204d8cf36934607707f82ba20af224026921335eb23d70b275be971"}
Jul 16 22:31:55.624241 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:55.624202    2566 generic.go:356] "Generic (PLEG): container finished" podID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerID="91e76c631204d8cf36934607707f82ba20af224026921335eb23d70b275be971" exitCode=6
Jul 16 22:31:55.624728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:55.624275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerDied","Data":"91e76c631204d8cf36934607707f82ba20af224026921335eb23d70b275be971"}
Jul 16 22:31:55.624728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:55.624367    2566 scope.go:117] "RemoveContainer" containerID="c20e4cdcb2d4ce4938cf24b5920cab8b2f8722d66c457d9a33a1ede1bc932f2c"
Jul 16 22:31:55.624844 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:55.624779    2566 scope.go:117] "RemoveContainer" containerID="91e76c631204d8cf36934607707f82ba20af224026921335eb23d70b275be971"
Jul 16 22:31:55.625047 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:31:55.625029    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nn88x_odh-ai-gateway-infra(489a29b3-bfcb-4db5-919e-f7d0f4f50880)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880"
Jul 16 22:31:55.689624 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:55.689589    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:56.301314 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.301278    2566 patch_prober.go:28] interesting pod/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:36402->10.132.0.85:15021: read: connection reset by peer" start-of-body=
Jul 16 22:31:56.301530 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.301354    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:36402->10.132.0.85:15021: read: connection reset by peer"
Jul 16 22:31:56.632953 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.632928    2566 generic.go:356] "Generic (PLEG): container finished" podID="42b83187-6428-4f07-bf99-265637affd0c" containerID="20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2" exitCode=0
Jul 16 22:31:56.633257 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.632978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerDied","Data":"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"}
Jul 16 22:31:56.633377 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.633361    2566 scope.go:117] "RemoveContainer" containerID="20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"
Jul 16 22:31:56.634495 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.634444    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:56.634582 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.634562    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:56.634644 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.634625    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:31:56.792920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.792603    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:56.896146 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.896049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca\") pod \"489a29b3-bfcb-4db5-919e-f7d0f4f50880\" (UID: \"489a29b3-bfcb-4db5-919e-f7d0f4f50880\") "
Jul 16 22:31:56.896444 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.896415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "489a29b3-bfcb-4db5-919e-f7d0f4f50880" (UID: "489a29b3-bfcb-4db5-919e-f7d0f4f50880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:56.896706 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:56.896592    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/489a29b3-bfcb-4db5-919e-f7d0f4f50880-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:31:57.644093 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.644041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x" event={"ID":"489a29b3-bfcb-4db5-919e-f7d0f4f50880","Type":"ContainerDied","Data":"d4d145dd16822a8152db4edb8316c7cae74b761ba1bd66241c7b46ce665113ed"}
Jul 16 22:31:57.644594 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.644132    2566 scope.go:117] "RemoveContainer" containerID="91e76c631204d8cf36934607707f82ba20af224026921335eb23d70b275be971"
Jul 16 22:31:57.644594 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.644254    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"
Jul 16 22:31:57.651219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.650499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerStarted","Data":"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"}
Jul 16 22:31:57.714095 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.714063    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:57.721323 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:57.721291    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nn88x"]
Jul 16 22:31:58.011385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:58.011349    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" path="/var/lib/kubelet/pods/489a29b3-bfcb-4db5-919e-f7d0f4f50880/volumes"
Jul 16 22:31:58.295408 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:58.295320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:31:58.295591 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:58.295565    2566 patch_prober.go:28] interesting pod/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Jul 16 22:31:58.295649 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:58.295628    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Jul 16 22:31:59.296155 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:59.296118    2566 patch_prober.go:28] interesting pod/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Jul 16 22:31:59.296572 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:31:59.296176    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Jul 16 22:32:00.295787 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:00.295747    2566 patch_prober.go:28] interesting pod/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Jul 16 22:32:00.295965 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:00.295813    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Jul 16 22:32:01.295792 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:01.295750    2566 patch_prober.go:28] interesting pod/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Jul 16 22:32:01.296193 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:01.295824    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Jul 16 22:32:02.299106 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:02.299063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:32:02.299581 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:02.299384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:32:02.300079 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:02.300059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:32:22.175917 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:22.175836    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:32:22.181983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:22.181956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:32:30.044300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.044253    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:32:30.044853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.044562    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" podUID="b2212878-597d-49bb-9980-25f7dea0505a" containerName="maas-api" containerID="cri-o://b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49" gracePeriod=30
Jul 16 22:32:30.297421 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.297359    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:32:30.422694 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.422648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") pod \"b2212878-597d-49bb-9980-25f7dea0505a\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") "
Jul 16 22:32:30.422888 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.422747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvkz7\" (UniqueName: \"kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7\") pod \"b2212878-597d-49bb-9980-25f7dea0505a\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") "
Jul 16 22:32:30.422888 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.422838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca\") pod \"b2212878-597d-49bb-9980-25f7dea0505a\" (UID: \"b2212878-597d-49bb-9980-25f7dea0505a\") "
Jul 16 22:32:30.423210 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.423186    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2212878-597d-49bb-9980-25f7dea0505a" (UID: "b2212878-597d-49bb-9980-25f7dea0505a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:30.424873 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.424844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7" (OuterVolumeSpecName: "kube-api-access-qvkz7") pod "b2212878-597d-49bb-9980-25f7dea0505a" (UID: "b2212878-597d-49bb-9980-25f7dea0505a"). InnerVolumeSpecName "kube-api-access-qvkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:32:30.425075 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.425050    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2212878-597d-49bb-9980-25f7dea0505a" (UID: "b2212878-597d-49bb-9980-25f7dea0505a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:32:30.523904 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.523870    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvkz7\" (UniqueName: \"kubernetes.io/projected/b2212878-597d-49bb-9980-25f7dea0505a-kube-api-access-qvkz7\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:32:30.523904 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.523898    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2212878-597d-49bb-9980-25f7dea0505a-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:32:30.523904 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.523910    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2212878-597d-49bb-9980-25f7dea0505a-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:32:30.838660 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.838625    2566 generic.go:356] "Generic (PLEG): container finished" podID="b2212878-597d-49bb-9980-25f7dea0505a" containerID="b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49" exitCode=0
Jul 16 22:32:30.838838 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.838670    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"
Jul 16 22:32:30.838838 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.838712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" event={"ID":"b2212878-597d-49bb-9980-25f7dea0505a","Type":"ContainerDied","Data":"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"}
Jul 16 22:32:30.838838 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.838794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd" event={"ID":"b2212878-597d-49bb-9980-25f7dea0505a","Type":"ContainerDied","Data":"755c59a2583cedabae5807bb3b8b7376d1932bd3e06dd1888948cf2335df9a17"}
Jul 16 22:32:30.838838 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.838822    2566 scope.go:117] "RemoveContainer" containerID="b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"
Jul 16 22:32:30.848333 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.848315    2566 scope.go:117] "RemoveContainer" containerID="b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"
Jul 16 22:32:30.848604 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:32:30.848586    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49\": container with ID starting with b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49 not found: ID does not exist" containerID="b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"
Jul 16 22:32:30.848653 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.848614    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49"} err="failed to get container status \"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49\": rpc error: code = NotFound desc = could not find container \"b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49\": container with ID starting with b2fbef17fc50580dfe2243452f6942341c5ab80f782e7a29cb020d37b4838a49 not found: ID does not exist"
Jul 16 22:32:30.862707 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.862687    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:32:30.866973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:30.866950    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-601103b4-68c8c445c9-w4mtd"]
Jul 16 22:32:32.010622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:32.010586    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2212878-597d-49bb-9980-25f7dea0505a" path="/var/lib/kubelet/pods/b2212878-597d-49bb-9980-25f7dea0505a/volumes"
Jul 16 22:32:35.358952 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.358919    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:35.359385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359371    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359386    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359399    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2212878-597d-49bb-9980-25f7dea0505a" containerName="maas-api"
Jul 16 22:32:35.359428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359404    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2212878-597d-49bb-9980-25f7dea0505a" containerName="maas-api"
Jul 16 22:32:35.359428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359416    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359425    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359439    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359445    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359472    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359478    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359564    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359575    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359584    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359593    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359600    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:32:35.359626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.359606    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2212878-597d-49bb-9980-25f7dea0505a" containerName="maas-api"
Jul 16 22:32:35.362725 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.362708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:35.365273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.365247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:32:35.370715 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.370418    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:35.471103 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.471071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hkflx\" (UID: \"321a8fe4-a448-47ab-9fda-34c411f0b97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:35.571772 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.571740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hkflx\" (UID: \"321a8fe4-a448-47ab-9fda-34c411f0b97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:35.572326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.572308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-hkflx\" (UID: \"321a8fe4-a448-47ab-9fda-34c411f0b97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:35.674888 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:35.674819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:36.035920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:36.035892    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:36.037145 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:32:36.037118    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321a8fe4_a448_47ab_9fda_34c411f0b97e.slice/crio-d729547201c55c18b1832e16461c4dbcbf294039551f4ab70cc240aaf60f24f9 WatchSource:0}: Error finding container d729547201c55c18b1832e16461c4dbcbf294039551f4ab70cc240aaf60f24f9: Status 404 returned error can't find the container with id d729547201c55c18b1832e16461c4dbcbf294039551f4ab70cc240aaf60f24f9
Jul 16 22:32:36.874232 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:36.874197    2566 generic.go:356] "Generic (PLEG): container finished" podID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerID="351a66a20287bc736fc3da0e5c7980ba6f36c860b2d3ed25bb1b24fe164891d3" exitCode=6
Jul 16 22:32:36.874598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:36.874281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" event={"ID":"321a8fe4-a448-47ab-9fda-34c411f0b97e","Type":"ContainerDied","Data":"351a66a20287bc736fc3da0e5c7980ba6f36c860b2d3ed25bb1b24fe164891d3"}
Jul 16 22:32:36.874598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:36.874346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" event={"ID":"321a8fe4-a448-47ab-9fda-34c411f0b97e","Type":"ContainerStarted","Data":"d729547201c55c18b1832e16461c4dbcbf294039551f4ab70cc240aaf60f24f9"}
Jul 16 22:32:36.874598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:36.874505    2566 scope.go:117] "RemoveContainer" containerID="351a66a20287bc736fc3da0e5c7980ba6f36c860b2d3ed25bb1b24fe164891d3"
Jul 16 22:32:37.881041 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:37.881007    2566 generic.go:356] "Generic (PLEG): container finished" podID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerID="e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5" exitCode=6
Jul 16 22:32:37.881526 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:37.881071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" event={"ID":"321a8fe4-a448-47ab-9fda-34c411f0b97e","Type":"ContainerDied","Data":"e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5"}
Jul 16 22:32:37.881526 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:37.881130    2566 scope.go:117] "RemoveContainer" containerID="351a66a20287bc736fc3da0e5c7980ba6f36c860b2d3ed25bb1b24fe164891d3"
Jul 16 22:32:37.881526 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:37.881373    2566 scope.go:117] "RemoveContainer" containerID="e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5"
Jul 16 22:32:37.881694 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:32:37.881643    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hkflx_odh-ai-gateway-infra(321a8fe4-a448-47ab-9fda-34c411f0b97e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e"
Jul 16 22:32:38.888535 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:38.888507    2566 scope.go:117] "RemoveContainer" containerID="e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5"
Jul 16 22:32:38.888891 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:32:38.888709    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hkflx_odh-ai-gateway-infra(321a8fe4-a448-47ab-9fda-34c411f0b97e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e"
Jul 16 22:32:54.004604 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:54.004574    2566 scope.go:117] "RemoveContainer" containerID="e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5"
Jul 16 22:32:54.980524 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:54.980425    2566 generic.go:356] "Generic (PLEG): container finished" podID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerID="4e3619edeeddb891a565592474bee6f33bb617af75a63aed5856cd870459f0f3" exitCode=6
Jul 16 22:32:54.980524 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:54.980496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" event={"ID":"321a8fe4-a448-47ab-9fda-34c411f0b97e","Type":"ContainerDied","Data":"4e3619edeeddb891a565592474bee6f33bb617af75a63aed5856cd870459f0f3"}
Jul 16 22:32:54.980719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:54.980555    2566 scope.go:117] "RemoveContainer" containerID="e790ac947c28e3c697eafbe72650432ca99791456e0229505d00a830ced2d6f5"
Jul 16 22:32:54.980785 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:54.980773    2566 scope.go:117] "RemoveContainer" containerID="4e3619edeeddb891a565592474bee6f33bb617af75a63aed5856cd870459f0f3"
Jul 16 22:32:54.980997 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:32:54.980975    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-hkflx_odh-ai-gateway-infra(321a8fe4-a448-47ab-9fda-34c411f0b97e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e"
Jul 16 22:32:56.011252 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.011224    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:56.148677 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.148651    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:56.253769 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.253731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca\") pod \"321a8fe4-a448-47ab-9fda-34c411f0b97e\" (UID: \"321a8fe4-a448-47ab-9fda-34c411f0b97e\") "
Jul 16 22:32:56.254059 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.254038    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "321a8fe4-a448-47ab-9fda-34c411f0b97e" (UID: "321a8fe4-a448-47ab-9fda-34c411f0b97e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:56.354674 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.354600    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/321a8fe4-a448-47ab-9fda-34c411f0b97e-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:32:56.994392 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.994364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"
Jul 16 22:32:56.994392 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.994373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx" event={"ID":"321a8fe4-a448-47ab-9fda-34c411f0b97e","Type":"ContainerDied","Data":"d729547201c55c18b1832e16461c4dbcbf294039551f4ab70cc240aaf60f24f9"}
Jul 16 22:32:56.994677 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:56.994434    2566 scope.go:117] "RemoveContainer" containerID="4e3619edeeddb891a565592474bee6f33bb617af75a63aed5856cd870459f0f3"
Jul 16 22:32:57.018105 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:57.018079    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:57.027911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:57.027886    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-hkflx"]
Jul 16 22:32:58.011304 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:32:58.011271    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" path="/var/lib/kubelet/pods/321a8fe4-a448-47ab-9fda-34c411f0b97e/volumes"
Jul 16 22:33:26.780308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:26.780272    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:33:26.780780 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:26.780532    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy" containerID="cri-o://524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef" gracePeriod=30
Jul 16 22:33:32.044516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.044486    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:33:32.082778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.082778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.082778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4czx\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.082990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data\") pod \"42b83187-6428-4f07-bf99-265637affd0c\" (UID: \"42b83187-6428-4f07-bf99-265637affd0c\") "
Jul 16 22:33:32.083078 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.083035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:32.083528 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.083513    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.084133 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.084084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:32.084275 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.084198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:32.084512 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.084489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:32.085083 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.085062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:32.085731 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.085693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:33:32.086567 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.086536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:33:32.086915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.086885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx" (OuterVolumeSpecName: "kube-api-access-q4czx") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "kube-api-access-q4czx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:33:32.087342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.087323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42b83187-6428-4f07-bf99-265637affd0c" (UID: "42b83187-6428-4f07-bf99-265637affd0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:32.184085 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184053    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b83187-6428-4f07-bf99-265637affd0c-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184085 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184079    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184085 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184088    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184085 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184095    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184103    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184112    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b83187-6428-4f07-bf99-265637affd0c-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184121    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4czx\" (UniqueName: \"kubernetes.io/projected/42b83187-6428-4f07-bf99-265637affd0c-kube-api-access-q4czx\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.184380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.184128    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b83187-6428-4f07-bf99-265637affd0c-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:32.188090 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.188063    2566 generic.go:356] "Generic (PLEG): container finished" podID="42b83187-6428-4f07-bf99-265637affd0c" containerID="524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef" exitCode=0
Jul 16 22:33:32.188160 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.188104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"
Jul 16 22:33:32.188160 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.188143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerDied","Data":"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"}
Jul 16 22:33:32.188228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.188192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr" event={"ID":"42b83187-6428-4f07-bf99-265637affd0c","Type":"ContainerDied","Data":"c3b0fc9c1e8e823208e5d3f3819fc9c1f3da0ef227b223b9d99812564a054cbf"}
Jul 16 22:33:32.188228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.188214    2566 scope.go:117] "RemoveContainer" containerID="524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"
Jul 16 22:33:32.198326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.198309    2566 scope.go:117] "RemoveContainer" containerID="20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"
Jul 16 22:33:32.206819 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.206801    2566 scope.go:117] "RemoveContainer" containerID="524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"
Jul 16 22:33:32.207080 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:32.207058    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef\": container with ID starting with 524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef not found: ID does not exist" containerID="524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"
Jul 16 22:33:32.207128 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.207089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef"} err="failed to get container status \"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef\": rpc error: code = NotFound desc = could not find container \"524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef\": container with ID starting with 524521167f03b398c74911f193bec09a5fb659811f506c4df0d40cb9b01a0bef not found: ID does not exist"
Jul 16 22:33:32.207128 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.207108    2566 scope.go:117] "RemoveContainer" containerID="20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"
Jul 16 22:33:32.207346 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:32.207327    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2\": container with ID starting with 20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2 not found: ID does not exist" containerID="20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"
Jul 16 22:33:32.207394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.207351    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2"} err="failed to get container status \"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2\": rpc error: code = NotFound desc = could not find container \"20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2\": container with ID starting with 20d0a938a4ac95c78a4b8648f860c282a3140bed18a5d790ce7b06c1fd5e82c2 not found: ID does not exist"
Jul 16 22:33:32.219620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.219594    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:33:32.230064 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:32.230042    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-601103b4-openshift-default-774bfbd6b7-v8ffr"]
Jul 16 22:33:34.011703 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:34.011668    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b83187-6428-4f07-bf99-265637affd0c" path="/var/lib/kubelet/pods/42b83187-6428-4f07-bf99-265637affd0c/volumes"
Jul 16 22:33:35.413315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413283    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:35.413755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413742    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.413755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413756    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413769    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413775    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413792    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413798    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413811    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="489a29b3-bfcb-4db5-919e-f7d0f4f50880" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413824    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413829    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413836    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.413840 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413841    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413904    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413911    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b83187-6428-4f07-bf99-265637affd0c" containerName="istio-proxy"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413916    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc401023-e241-4435-b359-2591779774ff" containerName="revoke-keys"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413925    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413933    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.414168 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.413940    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:35.418269 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.418251    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:35.420853 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.420836    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:33:35.425327 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.425297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:35.516773 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.516737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9\" (UID: \"64629a3c-c190-4fce-8aaf-d8fa0875b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:35.617670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.617641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9\" (UID: \"64629a3c-c190-4fce-8aaf-d8fa0875b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:35.618280 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.618261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9\" (UID: \"64629a3c-c190-4fce-8aaf-d8fa0875b7fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:35.730919 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:35.730824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:36.078188 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:36.078157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:36.080367 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:33:36.080340    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64629a3c_c190_4fce_8aaf_d8fa0875b7fb.slice/crio-a585313d5b57a6481c5dc69250770ea664b9c91c6de71d01baefc5a315819ccf WatchSource:0}: Error finding container a585313d5b57a6481c5dc69250770ea664b9c91c6de71d01baefc5a315819ccf: Status 404 returned error can't find the container with id a585313d5b57a6481c5dc69250770ea664b9c91c6de71d01baefc5a315819ccf
Jul 16 22:33:36.210839 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:36.210800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" event={"ID":"64629a3c-c190-4fce-8aaf-d8fa0875b7fb","Type":"ContainerStarted","Data":"a585313d5b57a6481c5dc69250770ea664b9c91c6de71d01baefc5a315819ccf"}
Jul 16 22:33:37.217710 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:37.217675    2566 generic.go:356] "Generic (PLEG): container finished" podID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerID="2377791626c42d86e4f55d9ce92091b2b79873ccd21f903d0629b8391a5f98c3" exitCode=6
Jul 16 22:33:37.218179 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:37.217722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" event={"ID":"64629a3c-c190-4fce-8aaf-d8fa0875b7fb","Type":"ContainerDied","Data":"2377791626c42d86e4f55d9ce92091b2b79873ccd21f903d0629b8391a5f98c3"}
Jul 16 22:33:37.218179 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:37.217990    2566 scope.go:117] "RemoveContainer" containerID="2377791626c42d86e4f55d9ce92091b2b79873ccd21f903d0629b8391a5f98c3"
Jul 16 22:33:38.224834 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.224744    2566 generic.go:356] "Generic (PLEG): container finished" podID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerID="9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4" exitCode=6
Jul 16 22:33:38.224834 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.224805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" event={"ID":"64629a3c-c190-4fce-8aaf-d8fa0875b7fb","Type":"ContainerDied","Data":"9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4"}
Jul 16 22:33:38.225259 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.224848    2566 scope.go:117] "RemoveContainer" containerID="2377791626c42d86e4f55d9ce92091b2b79873ccd21f903d0629b8391a5f98c3"
Jul 16 22:33:38.225259 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.225033    2566 scope.go:117] "RemoveContainer" containerID="9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4"
Jul 16 22:33:38.225326 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:38.225288    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9_odh-ai-gateway-infra(64629a3c-c190-4fce-8aaf-d8fa0875b7fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb"
Jul 16 22:33:38.444618 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.444585    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:38.445077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.445061    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:38.445120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.445079    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="321a8fe4-a448-47ab-9fda-34c411f0b97e" containerName="revoke-keys"
Jul 16 22:33:38.449561 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.449544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:38.460409 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.460379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:38.545523 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.545490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bkctp\" (UID: \"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:38.646817 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.646777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bkctp\" (UID: \"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:38.647443 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.647416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-bkctp\" (UID: \"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:38.760696 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.760653    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:38.901759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:38.901726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:38.904885 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:33:38.904856    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4299a9_9b0f_48b0_bd99_3a7a6ac4745b.slice/crio-38904d4ae3fa4f095448bb2389190f43a9f5936b77943810e2fa70551b52dae1 WatchSource:0}: Error finding container 38904d4ae3fa4f095448bb2389190f43a9f5936b77943810e2fa70551b52dae1: Status 404 returned error can't find the container with id 38904d4ae3fa4f095448bb2389190f43a9f5936b77943810e2fa70551b52dae1
Jul 16 22:33:39.233321 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:39.233294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" event={"ID":"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b","Type":"ContainerStarted","Data":"38904d4ae3fa4f095448bb2389190f43a9f5936b77943810e2fa70551b52dae1"}
Jul 16 22:33:39.235280 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:39.235249    2566 scope.go:117] "RemoveContainer" containerID="9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4"
Jul 16 22:33:39.235482 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:39.235439    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9_odh-ai-gateway-infra(64629a3c-c190-4fce-8aaf-d8fa0875b7fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb"
Jul 16 22:33:40.242207 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:40.242173    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerID="f37a9964ca726ce80c0af84c3ece2f3564a42e05f09d31aab792c5fe051c509e" exitCode=6
Jul 16 22:33:40.242678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:40.242236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" event={"ID":"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b","Type":"ContainerDied","Data":"f37a9964ca726ce80c0af84c3ece2f3564a42e05f09d31aab792c5fe051c509e"}
Jul 16 22:33:40.242678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:40.242420    2566 scope.go:117] "RemoveContainer" containerID="f37a9964ca726ce80c0af84c3ece2f3564a42e05f09d31aab792c5fe051c509e"
Jul 16 22:33:41.249413 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:41.249381    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerID="344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b" exitCode=6
Jul 16 22:33:41.249901 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:41.249487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" event={"ID":"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b","Type":"ContainerDied","Data":"344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b"}
Jul 16 22:33:41.249901 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:41.249545    2566 scope.go:117] "RemoveContainer" containerID="f37a9964ca726ce80c0af84c3ece2f3564a42e05f09d31aab792c5fe051c509e"
Jul 16 22:33:41.249901 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:41.249726    2566 scope.go:117] "RemoveContainer" containerID="344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b"
Jul 16 22:33:41.250061 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:41.249994    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bkctp_odh-ai-gateway-infra(2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b"
Jul 16 22:33:42.257499 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:42.257447    2566 scope.go:117] "RemoveContainer" containerID="344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b"
Jul 16 22:33:42.257915 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:42.257730    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bkctp_odh-ai-gateway-infra(2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b"
Jul 16 22:33:50.005387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:50.005356    2566 scope.go:117] "RemoveContainer" containerID="9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4"
Jul 16 22:33:51.308912 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:51.308872    2566 generic.go:356] "Generic (PLEG): container finished" podID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerID="209d30755d76344b7d0dd19eb17f28c29e054a18904d5b6eac05833af76fe09c" exitCode=6
Jul 16 22:33:51.309307 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:51.308943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" event={"ID":"64629a3c-c190-4fce-8aaf-d8fa0875b7fb","Type":"ContainerDied","Data":"209d30755d76344b7d0dd19eb17f28c29e054a18904d5b6eac05833af76fe09c"}
Jul 16 22:33:51.309307 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:51.309004    2566 scope.go:117] "RemoveContainer" containerID="9b21ca19c4c0456d4724b6b195aa18205df03d4e374e78df310e07e6bdd9a7f4"
Jul 16 22:33:51.309307 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:51.309268    2566 scope.go:117] "RemoveContainer" containerID="209d30755d76344b7d0dd19eb17f28c29e054a18904d5b6eac05833af76fe09c"
Jul 16 22:33:51.309551 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:51.309531    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9_odh-ai-gateway-infra(64629a3c-c190-4fce-8aaf-d8fa0875b7fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb"
Jul 16 22:33:52.334967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:52.334933    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:52.465823 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:52.465799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:52.583433 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:52.583401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca\") pod \"64629a3c-c190-4fce-8aaf-d8fa0875b7fb\" (UID: \"64629a3c-c190-4fce-8aaf-d8fa0875b7fb\") "
Jul 16 22:33:52.583793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:52.583767    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64629a3c-c190-4fce-8aaf-d8fa0875b7fb" (UID: "64629a3c-c190-4fce-8aaf-d8fa0875b7fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:52.684487 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:52.684362    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64629a3c-c190-4fce-8aaf-d8fa0875b7fb-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:53.322300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:53.322271    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"
Jul 16 22:33:53.322538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:53.322316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9" event={"ID":"64629a3c-c190-4fce-8aaf-d8fa0875b7fb","Type":"ContainerDied","Data":"a585313d5b57a6481c5dc69250770ea664b9c91c6de71d01baefc5a315819ccf"}
Jul 16 22:33:53.322538 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:53.322398    2566 scope.go:117] "RemoveContainer" containerID="209d30755d76344b7d0dd19eb17f28c29e054a18904d5b6eac05833af76fe09c"
Jul 16 22:33:53.347530 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:53.347492    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:53.354102 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:53.354081    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-nnqt9"]
Jul 16 22:33:54.004973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:54.004940    2566 scope.go:117] "RemoveContainer" containerID="344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b"
Jul 16 22:33:54.011621 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:54.011599    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" path="/var/lib/kubelet/pods/64629a3c-c190-4fce-8aaf-d8fa0875b7fb/volumes"
Jul 16 22:33:55.343237 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:55.343204    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerID="9b3d46540905d3531f586c76ea14244366a62dfbd35ee3630ece033f30c435dc" exitCode=6
Jul 16 22:33:55.343652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:55.343277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" event={"ID":"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b","Type":"ContainerDied","Data":"9b3d46540905d3531f586c76ea14244366a62dfbd35ee3630ece033f30c435dc"}
Jul 16 22:33:55.343652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:55.343371    2566 scope.go:117] "RemoveContainer" containerID="344e808053a9638a4be9b956c4ea55fdfab68016b31b8be0daee3e3541a4782b"
Jul 16 22:33:55.343782 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:55.343677    2566 scope.go:117] "RemoveContainer" containerID="9b3d46540905d3531f586c76ea14244366a62dfbd35ee3630ece033f30c435dc"
Jul 16 22:33:55.343970 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:55.343946    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-bkctp_odh-ai-gateway-infra(2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b"
Jul 16 22:33:56.375321 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:56.375285    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:56.522719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:56.522686    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:56.621336 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:56.621305    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca\") pod \"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b\" (UID: \"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b\") "
Jul 16 22:33:56.621676 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:56.621653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" (UID: "2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:56.723102 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:56.723021    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:33:57.162830 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.162796    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:33:57.163234 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163221    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163235    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163257    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163269    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163278 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163274    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163296    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163305    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163310    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163367    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163383    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:33:57.163559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.163392    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:33:57.166618 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.166598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:33:57.178939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.178905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:33:57.329160 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.329121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2\" (UID: \"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:33:57.356976 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.356934    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"
Jul 16 22:33:57.357159 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.356982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp" event={"ID":"2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b","Type":"ContainerDied","Data":"38904d4ae3fa4f095448bb2389190f43a9f5936b77943810e2fa70551b52dae1"}
Jul 16 22:33:57.357159 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.357044    2566 scope.go:117] "RemoveContainer" containerID="9b3d46540905d3531f586c76ea14244366a62dfbd35ee3630ece033f30c435dc"
Jul 16 22:33:57.393369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.393333    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:57.396781 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.396759    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-bkctp"]
Jul 16 22:33:57.430625 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.430538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2\" (UID: \"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:33:57.431176 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.431156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2\" (UID: \"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:33:57.477085 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.477052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:33:57.622790 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:57.622762    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:33:57.624019 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:33:57.623992    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51fa9b_06fc_4383_a5e7_762ecfdd58a4.slice/crio-067a640fd14c06b2bd1ef0947cbc009408333251d2cea3fd36f1eb46a09f3aa3 WatchSource:0}: Error finding container 067a640fd14c06b2bd1ef0947cbc009408333251d2cea3fd36f1eb46a09f3aa3: Status 404 returned error can't find the container with id 067a640fd14c06b2bd1ef0947cbc009408333251d2cea3fd36f1eb46a09f3aa3
Jul 16 22:33:58.011944 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:58.011907    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" path="/var/lib/kubelet/pods/2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b/volumes"
Jul 16 22:33:58.365311 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:58.365273    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerID="a3ca9ca186cd978f54d731881615c0c7d3024960bc571a70b81f13eb5924cb94" exitCode=6
Jul 16 22:33:58.365486 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:58.365332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerDied","Data":"a3ca9ca186cd978f54d731881615c0c7d3024960bc571a70b81f13eb5924cb94"}
Jul 16 22:33:58.365486 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:58.365373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerStarted","Data":"067a640fd14c06b2bd1ef0947cbc009408333251d2cea3fd36f1eb46a09f3aa3"}
Jul 16 22:33:58.365578 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:58.365569    2566 scope.go:117] "RemoveContainer" containerID="a3ca9ca186cd978f54d731881615c0c7d3024960bc571a70b81f13eb5924cb94"
Jul 16 22:33:59.373162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:59.373122    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerID="4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c" exitCode=6
Jul 16 22:33:59.373592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:59.373215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerDied","Data":"4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c"}
Jul 16 22:33:59.373592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:59.373275    2566 scope.go:117] "RemoveContainer" containerID="a3ca9ca186cd978f54d731881615c0c7d3024960bc571a70b81f13eb5924cb94"
Jul 16 22:33:59.373592 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:33:59.373448    2566 scope.go:117] "RemoveContainer" containerID="4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c"
Jul 16 22:33:59.373730 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:33:59.373711    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2_odh-ai-gateway-infra(9c51fa9b-06fc-4383-a5e7-762ecfdd58a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4"
Jul 16 22:34:00.380869 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:00.380834    2566 scope.go:117] "RemoveContainer" containerID="4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c"
Jul 16 22:34:00.381259 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:34:00.381023    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2_odh-ai-gateway-infra(9c51fa9b-06fc-4383-a5e7-762ecfdd58a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4"
Jul 16 22:34:15.005250 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:15.005207    2566 scope.go:117] "RemoveContainer" containerID="4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c"
Jul 16 22:34:15.475618 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:15.475582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerStarted","Data":"28fda95fe205513dcdf2ddef13e5b0edfef6f9ad6f3b75a0e95c18a7d5451d42"}
Jul 16 22:34:15.494308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:15.494249    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" podStartSLOduration=18.494231038 podStartE2EDuration="18.494231038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:15.491627473 +0000 UTC m=+2814.001494570" watchObservedRunningTime="2026-07-16 22:34:15.494231038 +0000 UTC m=+2814.004098120"
Jul 16 22:34:16.483857 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:16.483821    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerID="28fda95fe205513dcdf2ddef13e5b0edfef6f9ad6f3b75a0e95c18a7d5451d42" exitCode=6
Jul 16 22:34:16.484374 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:16.483894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerDied","Data":"28fda95fe205513dcdf2ddef13e5b0edfef6f9ad6f3b75a0e95c18a7d5451d42"}
Jul 16 22:34:16.484374 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:16.483976    2566 scope.go:117] "RemoveContainer" containerID="4217b0586b10c6039373d0427674621e5e6fca3c17dc67949cb31e75832d3d9c"
Jul 16 22:34:16.484374 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:16.484226    2566 scope.go:117] "RemoveContainer" containerID="28fda95fe205513dcdf2ddef13e5b0edfef6f9ad6f3b75a0e95c18a7d5451d42"
Jul 16 22:34:16.484655 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:34:16.484553    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2_odh-ai-gateway-infra(9c51fa9b-06fc-4383-a5e7-762ecfdd58a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4"
Jul 16 22:34:16.510101 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:16.510067    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:34:17.637740 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:17.637708    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:34:17.726103 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:17.726063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca\") pod \"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4\" (UID: \"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4\") "
Jul 16 22:34:17.726497 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:17.726440    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" (UID: "9c51fa9b-06fc-4383-a5e7-762ecfdd58a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:17.827223 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:17.827129    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:34:18.497787 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:18.497752    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"
Jul 16 22:34:18.498002 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:18.497792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2" event={"ID":"9c51fa9b-06fc-4383-a5e7-762ecfdd58a4","Type":"ContainerDied","Data":"067a640fd14c06b2bd1ef0947cbc009408333251d2cea3fd36f1eb46a09f3aa3"}
Jul 16 22:34:18.498002 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:18.497854    2566 scope.go:117] "RemoveContainer" containerID="28fda95fe205513dcdf2ddef13e5b0edfef6f9ad6f3b75a0e95c18a7d5451d42"
Jul 16 22:34:18.518109 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:18.518072    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:34:18.524446 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:18.524414    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-xntx2"]
Jul 16 22:34:20.011342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:20.011307    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" path="/var/lib/kubelet/pods/9c51fa9b-06fc-4383-a5e7-762ecfdd58a4/volumes"
Jul 16 22:34:22.677653 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.677562    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:34:22.678097 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678080    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.678142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678099    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.678142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.678142 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678120    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.678240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678213    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.678240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678223    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64629a3c-c190-4fce-8aaf-d8fa0875b7fb" containerName="revoke-keys"
Jul 16 22:34:22.678240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678233    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4299a9-9b0f-48b0-bd99-3a7a6ac4745b" containerName="revoke-keys"
Jul 16 22:34:22.678342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.678241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:22.682943 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.682917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.686090 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.686066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c36d62-openshift-default-dockercfg-5g5ng\""
Jul 16 22:34:22.705660 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.705629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:34:22.772687 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.772896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.772896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.772896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.772896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clbs\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.772896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.773183 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.773183 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.772953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.773183 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.773002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.873790 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clbs\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874003 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.873973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874345 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874345 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874345 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874345 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874555 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874660 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.874717 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.874673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.876390 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.876372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.876554 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.876538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.895050 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.895019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.895598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.895574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clbs\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs\") pod \"e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") " pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:22.993678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:22.993637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:23.168434 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.168405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:34:23.171486 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:34:23.171425    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e04820_1f71_4d05_b0ee_6fabf78b415c.slice/crio-53fb822d7ad51d6827cbe088f4277742f855b622e99343f9c9a1f5a6ce147634 WatchSource:0}: Error finding container 53fb822d7ad51d6827cbe088f4277742f855b622e99343f9c9a1f5a6ce147634: Status 404 returned error can't find the container with id 53fb822d7ad51d6827cbe088f4277742f855b622e99343f9c9a1f5a6ce147634
Jul 16 22:34:23.173664 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.173630    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:23.173733 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.173702    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:23.173781 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.173737    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:23.540271 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.540218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerStarted","Data":"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"}
Jul 16 22:34:23.540524 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.540380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerStarted","Data":"53fb822d7ad51d6827cbe088f4277742f855b622e99343f9c9a1f5a6ce147634"}
Jul 16 22:34:23.994569 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:23.994531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:24.999023 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:24.998986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:25.023054 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.022995    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podStartSLOduration=3.022977078 podStartE2EDuration="3.022977078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:23.576226668 +0000 UTC m=+2822.086093751" watchObservedRunningTime="2026-07-16 22:34:25.022977078 +0000 UTC m=+2823.532844251"
Jul 16 22:34:25.551859 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.551821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:25.552747 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.552726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:25.816681 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.816595    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:34:25.817100 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.817086    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:25.817144 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.817102    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:25.817184 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.817177    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51fa9b-06fc-4383-a5e7-762ecfdd58a4" containerName="revoke-keys"
Jul 16 22:34:25.820694 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.820674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:25.825143 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.825119    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c36d62-serving-cert\""
Jul 16 22:34:25.826492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.826447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:34:25.837429 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.837402    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:34:25.902931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.902893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlm6\" (UniqueName: \"kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:25.903126 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.902959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:25.903126 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:25.902990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.004179 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.004140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlm6\" (UniqueName: \"kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.004695 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.004201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.004695 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.004236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.005027 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.004996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.007099 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.007075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.015556 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.015528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlm6\" (UniqueName: \"kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6\") pod \"maas-api-e2e-inf-a-c36d62-8999876cb-sghjr\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.132938 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.132838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:26.291532 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.291499    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:34:26.292052 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:34:26.292026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fab4df_65cf_4394_abd1_b99ac91e9012.slice/crio-592e8b7713f075a06a7a13eaf41070b2ae42e19340a696ee4aee284d81b036d5 WatchSource:0}: Error finding container 592e8b7713f075a06a7a13eaf41070b2ae42e19340a696ee4aee284d81b036d5: Status 404 returned error can't find the container with id 592e8b7713f075a06a7a13eaf41070b2ae42e19340a696ee4aee284d81b036d5
Jul 16 22:34:26.558064 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:26.558028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" event={"ID":"b6fab4df-65cf-4394-abd1-b99ac91e9012","Type":"ContainerStarted","Data":"592e8b7713f075a06a7a13eaf41070b2ae42e19340a696ee4aee284d81b036d5"}
Jul 16 22:34:27.565141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:27.565100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" event={"ID":"b6fab4df-65cf-4394-abd1-b99ac91e9012","Type":"ContainerStarted","Data":"13af5fa0bf28d958807ece1600d8f6aeca38ae3d324ce9673d6fb89fd1018449"}
Jul 16 22:34:27.565585 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:27.565172    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:27.586024 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:27.585962    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" podStartSLOduration=2.103718186 podStartE2EDuration="2.585942906s" totalImagesPullingTime="482.22472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:27.58369405 +0000 UTC m=+2826.093561126" watchObservedRunningTime="2026-07-16 22:34:27.585942906 +0000 UTC m=+2826.095809988"
Jul 16 22:34:33.576927 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:33.576892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:34:36.056507 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.050416    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:34:36.058804 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.058775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:34:36.058939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.058885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.061414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.061394    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cca4a2-serving-cert\""
Jul 16 22:34:36.210445 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.210406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.210623 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.210494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.210623 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.210596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnz5\" (UniqueName: \"kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.312178 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.312095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.312178 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.312139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.312419 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.312189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnz5\" (UniqueName: \"kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.312419 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:34:36.312320    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-serving-cert: secret "maas-api-e2e-inf-b-cca4a2-serving-cert" not found
Jul 16 22:34:36.312419 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:34:36.312400    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls podName:cfb716a3-e7ff-4739-95a9-9c6454f71f12 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:36.812382223 +0000 UTC m=+2835.322249288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls") pod "maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" (UID: "cfb716a3-e7ff-4739-95a9-9c6454f71f12") : secret "maas-api-e2e-inf-b-cca4a2-serving-cert" not found
Jul 16 22:34:36.312802 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.312783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.321280 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.321253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnz5\" (UniqueName: \"kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.817740 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.817701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.820522 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.820495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") pod \"maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:36.972172 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:36.972135    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:37.143165 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:37.143119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:34:37.156132 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:34:37.156103    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb716a3_e7ff_4739_95a9_9c6454f71f12.slice/crio-8bb8152f2852c3baab7fafa5ebdf44476119ce43c86dc9e8b8004598ef4237c8 WatchSource:0}: Error finding container 8bb8152f2852c3baab7fafa5ebdf44476119ce43c86dc9e8b8004598ef4237c8: Status 404 returned error can't find the container with id 8bb8152f2852c3baab7fafa5ebdf44476119ce43c86dc9e8b8004598ef4237c8
Jul 16 22:34:37.619734 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:37.619690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" event={"ID":"cfb716a3-e7ff-4739-95a9-9c6454f71f12","Type":"ContainerStarted","Data":"8bb8152f2852c3baab7fafa5ebdf44476119ce43c86dc9e8b8004598ef4237c8"}
Jul 16 22:34:38.636871 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:38.636807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" event={"ID":"cfb716a3-e7ff-4739-95a9-9c6454f71f12","Type":"ContainerStarted","Data":"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"}
Jul 16 22:34:38.637722 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:38.637696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:38.657663 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:38.657617    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" podStartSLOduration=2.883398326 podStartE2EDuration="3.657602722s" totalImagesPullingTime="774.204396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:38.655487431 +0000 UTC m=+2837.165354512" watchObservedRunningTime="2026-07-16 22:34:38.657602722 +0000 UTC m=+2837.167469855"
Jul 16 22:34:41.593625 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:41.593577    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 22:34:41.594017 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:41.593650    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:34:42.664354 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.664316    2566 generic.go:356] "Generic (PLEG): container finished" podID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerID="82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b" exitCode=0
Jul 16 22:34:42.664886 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.664390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerDied","Data":"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"}
Jul 16 22:34:42.664886 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.664835    2566 scope.go:117] "RemoveContainer" containerID="82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"
Jul 16 22:34:42.665850 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.665822    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:42.665937 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.665888    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:42.665937 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:42.665920    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 16 22:34:43.677205 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:43.677162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerStarted","Data":"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"}
Jul 16 22:34:43.994434 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:43.994390    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:43.994702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:43.994680    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Jul 16 22:34:43.994772 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:43.994727    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Jul 16 22:34:44.994493 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:44.994432    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Jul 16 22:34:44.994893 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:44.994519    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Jul 16 22:34:45.657077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:45.657041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:34:45.995062 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:45.995022    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Jul 16 22:34:45.995549 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:45.995085    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Jul 16 22:34:46.994915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:46.994872    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Jul 16 22:34:46.995108 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:46.994954    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Jul 16 22:34:47.998236 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:47.998202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:47.998734 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:47.998578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:34:47.999537 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:34:47.999515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:35:55.859343 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:55.859290    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:35:55.859825 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:55.859539    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" podUID="b6fab4df-65cf-4394-abd1-b99ac91e9012" containerName="maas-api" containerID="cri-o://13af5fa0bf28d958807ece1600d8f6aeca38ae3d324ce9673d6fb89fd1018449" gracePeriod=30
Jul 16 22:35:56.089371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.089334    2566 generic.go:356] "Generic (PLEG): container finished" podID="b6fab4df-65cf-4394-abd1-b99ac91e9012" containerID="13af5fa0bf28d958807ece1600d8f6aeca38ae3d324ce9673d6fb89fd1018449" exitCode=0
Jul 16 22:35:56.089601 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.089531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" event={"ID":"b6fab4df-65cf-4394-abd1-b99ac91e9012","Type":"ContainerDied","Data":"13af5fa0bf28d958807ece1600d8f6aeca38ae3d324ce9673d6fb89fd1018449"}
Jul 16 22:35:56.136328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.136296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:35:56.223141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.223096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls\") pod \"b6fab4df-65cf-4394-abd1-b99ac91e9012\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") "
Jul 16 22:35:56.223330 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.223224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqlm6\" (UniqueName: \"kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6\") pod \"b6fab4df-65cf-4394-abd1-b99ac91e9012\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") "
Jul 16 22:35:56.223330 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.223295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca\") pod \"b6fab4df-65cf-4394-abd1-b99ac91e9012\" (UID: \"b6fab4df-65cf-4394-abd1-b99ac91e9012\") "
Jul 16 22:35:56.223774 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.223742    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b6fab4df-65cf-4394-abd1-b99ac91e9012" (UID: "b6fab4df-65cf-4394-abd1-b99ac91e9012"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:56.225315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.225280    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b6fab4df-65cf-4394-abd1-b99ac91e9012" (UID: "b6fab4df-65cf-4394-abd1-b99ac91e9012"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:35:56.225531 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.225500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6" (OuterVolumeSpecName: "kube-api-access-vqlm6") pod "b6fab4df-65cf-4394-abd1-b99ac91e9012" (UID: "b6fab4df-65cf-4394-abd1-b99ac91e9012"). InnerVolumeSpecName "kube-api-access-vqlm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:35:56.324616 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.324567    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqlm6\" (UniqueName: \"kubernetes.io/projected/b6fab4df-65cf-4394-abd1-b99ac91e9012-kube-api-access-vqlm6\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:35:56.324616 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.324614    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6fab4df-65cf-4394-abd1-b99ac91e9012-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:35:56.324817 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:56.324629    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6fab4df-65cf-4394-abd1-b99ac91e9012-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:35:57.096728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:57.096701    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"
Jul 16 22:35:57.097156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:57.096701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr" event={"ID":"b6fab4df-65cf-4394-abd1-b99ac91e9012","Type":"ContainerDied","Data":"592e8b7713f075a06a7a13eaf41070b2ae42e19340a696ee4aee284d81b036d5"}
Jul 16 22:35:57.097156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:57.096842    2566 scope.go:117] "RemoveContainer" containerID="13af5fa0bf28d958807ece1600d8f6aeca38ae3d324ce9673d6fb89fd1018449"
Jul 16 22:35:57.121783 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:57.121745    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:35:57.129037 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:57.128994    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c36d62-8999876cb-sghjr"]
Jul 16 22:35:58.011829 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:35:58.011792    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6fab4df-65cf-4394-abd1-b99ac91e9012" path="/var/lib/kubelet/pods/b6fab4df-65cf-4394-abd1-b99ac91e9012/volumes"
Jul 16 22:36:11.121568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:11.121528    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:36:11.121985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:11.121764    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy" containerID="cri-o://9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681" gracePeriod=30
Jul 16 22:36:16.393310 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.393237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:36:16.422052 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422023    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422052 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422300 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422205    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422547 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422547 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8clbs\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422547 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy\") pod \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\" (UID: \"f6e04820-1f71-4d05-b0ee-6fabf78b415c\") "
Jul 16 22:36:16.422547 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:16.423240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.422949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:16.423240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.423214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:16.423427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.423289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:16.423427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.423318    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.423732 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.423705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:16.425269 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.425234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:16.425269 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.425258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:16.425445 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.425421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs" (OuterVolumeSpecName: "kube-api-access-8clbs") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "kube-api-access-8clbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:16.425902 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.425882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6e04820-1f71-4d05-b0ee-6fabf78b415c" (UID: "f6e04820-1f71-4d05-b0ee-6fabf78b415c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:36:16.524082 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524041    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8clbs\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-kube-api-access-8clbs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524082 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524076    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-envoy\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524082 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524091    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-data\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524102    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-token\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524113    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istiod-ca-cert\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524126    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-workload-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524136    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e04820-1f71-4d05-b0ee-6fabf78b415c-credential-socket\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:16.524351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:16.524147    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e04820-1f71-4d05-b0ee-6fabf78b415c-istio-podinfo\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:17.216212 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.216169    2566 generic.go:356] "Generic (PLEG): container finished" podID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerID="9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681" exitCode=0
Jul 16 22:36:17.216394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.216225    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"
Jul 16 22:36:17.216394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.216242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerDied","Data":"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"}
Jul 16 22:36:17.216394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.216295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4" event={"ID":"f6e04820-1f71-4d05-b0ee-6fabf78b415c","Type":"ContainerDied","Data":"53fb822d7ad51d6827cbe088f4277742f855b622e99343f9c9a1f5a6ce147634"}
Jul 16 22:36:17.216394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.216319    2566 scope.go:117] "RemoveContainer" containerID="9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"
Jul 16 22:36:17.226556 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.226536    2566 scope.go:117] "RemoveContainer" containerID="82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"
Jul 16 22:36:17.235328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.235305    2566 scope.go:117] "RemoveContainer" containerID="9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"
Jul 16 22:36:17.235651 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:17.235632    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681\": container with ID starting with 9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681 not found: ID does not exist" containerID="9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"
Jul 16 22:36:17.235706 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.235660    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681"} err="failed to get container status \"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681\": rpc error: code = NotFound desc = could not find container \"9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681\": container with ID starting with 9b416344d53f2cf22769b68da0e16fdd0ecb39e3a1fa05fcfb3ec0dfeb148681 not found: ID does not exist"
Jul 16 22:36:17.235706 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.235683    2566 scope.go:117] "RemoveContainer" containerID="82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"
Jul 16 22:36:17.235903 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:17.235887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b\": container with ID starting with 82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b not found: ID does not exist" containerID="82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"
Jul 16 22:36:17.235949 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.235906    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b"} err="failed to get container status \"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b\": rpc error: code = NotFound desc = could not find container \"82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b\": container with ID starting with 82c60d58b141d70a56e210355ab1ee25d9675dcbffe27d089b57543ab0586a3b not found: ID does not exist"
Jul 16 22:36:17.241135 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.241110    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:36:17.245207 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:17.245180    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c36d62-openshift-default-5fc895f64f-jgxk4"]
Jul 16 22:36:18.011613 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:18.011568    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" path="/var/lib/kubelet/pods/f6e04820-1f71-4d05-b0ee-6fabf78b415c/volumes"
Jul 16 22:36:21.896543 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:21.896504    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:36:21.897156 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:21.896786    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" podUID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" containerName="maas-api" containerID="cri-o://40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683" gracePeriod=30
Jul 16 22:36:23.184096 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.184066    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:36:23.252721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.252670    2566 generic.go:356] "Generic (PLEG): container finished" podID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" containerID="40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683" exitCode=0
Jul 16 22:36:23.252721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.252720    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"
Jul 16 22:36:23.252983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.252735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" event={"ID":"cfb716a3-e7ff-4739-95a9-9c6454f71f12","Type":"ContainerDied","Data":"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"}
Jul 16 22:36:23.252983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.252796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x" event={"ID":"cfb716a3-e7ff-4739-95a9-9c6454f71f12","Type":"ContainerDied","Data":"8bb8152f2852c3baab7fafa5ebdf44476119ce43c86dc9e8b8004598ef4237c8"}
Jul 16 22:36:23.252983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.252822    2566 scope.go:117] "RemoveContainer" containerID="40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"
Jul 16 22:36:23.265532 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.265504    2566 scope.go:117] "RemoveContainer" containerID="40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"
Jul 16 22:36:23.265919 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:23.265894    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683\": container with ID starting with 40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683 not found: ID does not exist" containerID="40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"
Jul 16 22:36:23.265974 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.265932    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683"} err="failed to get container status \"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683\": rpc error: code = NotFound desc = could not find container \"40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683\": container with ID starting with 40d19b7abb9353a3c4a96ed3c4f7896ae3b355f6a103fa717c0fbc6a7e8f5683 not found: ID does not exist"
Jul 16 22:36:23.292308 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.292265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca\") pod \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") "
Jul 16 22:36:23.292511 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.292424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") pod \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") "
Jul 16 22:36:23.292511 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.292444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfnz5\" (UniqueName: \"kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5\") pod \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\" (UID: \"cfb716a3-e7ff-4739-95a9-9c6454f71f12\") "
Jul 16 22:36:23.292746 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.292719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfb716a3-e7ff-4739-95a9-9c6454f71f12" (UID: "cfb716a3-e7ff-4739-95a9-9c6454f71f12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:23.294634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.294606    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfb716a3-e7ff-4739-95a9-9c6454f71f12" (UID: "cfb716a3-e7ff-4739-95a9-9c6454f71f12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:23.294634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.294614    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5" (OuterVolumeSpecName: "kube-api-access-jfnz5") pod "cfb716a3-e7ff-4739-95a9-9c6454f71f12" (UID: "cfb716a3-e7ff-4739-95a9-9c6454f71f12"). InnerVolumeSpecName "kube-api-access-jfnz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:23.393533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.393446    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb716a3-e7ff-4739-95a9-9c6454f71f12-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:23.393533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.393529    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnz5\" (UniqueName: \"kubernetes.io/projected/cfb716a3-e7ff-4739-95a9-9c6454f71f12-kube-api-access-jfnz5\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:23.393533 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.393539    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb716a3-e7ff-4739-95a9-9c6454f71f12-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:23.575524 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.575482    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:36:23.579174 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:23.579142    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cca4a2-84c558f875-8rf6x"]
Jul 16 22:36:24.011677 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:24.011639    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" path="/var/lib/kubelet/pods/cfb716a3-e7ff-4739-95a9-9c6454f71f12/volumes"
Jul 16 22:36:25.627595 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.627552    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:25.628171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628149    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" containerName="maas-api"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" containerName="maas-api"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628189    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628198    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628225    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6fab4df-65cf-4394-abd1-b99ac91e9012" containerName="maas-api"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6fab4df-65cf-4394-abd1-b99ac91e9012" containerName="maas-api"
Jul 16 22:36:25.628261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628258    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628382    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628396    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e04820-1f71-4d05-b0ee-6fabf78b415c" containerName="istio-proxy"
Jul 16 22:36:25.628590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628407    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6fab4df-65cf-4394-abd1-b99ac91e9012" containerName="maas-api"
Jul 16 22:36:25.628590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.628425    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb716a3-e7ff-4739-95a9-9c6454f71f12" containerName="maas-api"
Jul 16 22:36:25.632741 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.632720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:25.636581 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.636563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:36:25.645329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.645304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:25.716842 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.716787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx\" (UID: \"712a36cd-8af7-43bd-8c8b-14619d1d0e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:25.817567 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.817531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx\" (UID: \"712a36cd-8af7-43bd-8c8b-14619d1d0e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:25.818208 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.818180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx\" (UID: \"712a36cd-8af7-43bd-8c8b-14619d1d0e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:25.943426 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:25.943315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:26.087811 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:26.087780    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:26.090620 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:36:26.090576    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712a36cd_8af7_43bd_8c8b_14619d1d0e0b.slice/crio-75940c6de5f0b7469c382d43d766016883ce5b73ce4b90f80a78f4ececfb3b27 WatchSource:0}: Error finding container 75940c6de5f0b7469c382d43d766016883ce5b73ce4b90f80a78f4ececfb3b27: Status 404 returned error can't find the container with id 75940c6de5f0b7469c382d43d766016883ce5b73ce4b90f80a78f4ececfb3b27
Jul 16 22:36:26.271772 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:26.271729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerStarted","Data":"75940c6de5f0b7469c382d43d766016883ce5b73ce4b90f80a78f4ececfb3b27"}
Jul 16 22:36:27.279652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:27.279617    2566 generic.go:356] "Generic (PLEG): container finished" podID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerID="80e89129487b39e5f32c55195a6c5b4aaa5154d276d01e6dff5dbdcc2607713b" exitCode=6
Jul 16 22:36:27.280013 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:27.279706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerDied","Data":"80e89129487b39e5f32c55195a6c5b4aaa5154d276d01e6dff5dbdcc2607713b"}
Jul 16 22:36:27.280013 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:27.279857    2566 scope.go:117] "RemoveContainer" containerID="80e89129487b39e5f32c55195a6c5b4aaa5154d276d01e6dff5dbdcc2607713b"
Jul 16 22:36:28.287163 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.287130    2566 generic.go:356] "Generic (PLEG): container finished" podID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerID="d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4" exitCode=6
Jul 16 22:36:28.287608 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.287196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerDied","Data":"d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4"}
Jul 16 22:36:28.287608 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.287265    2566 scope.go:117] "RemoveContainer" containerID="80e89129487b39e5f32c55195a6c5b4aaa5154d276d01e6dff5dbdcc2607713b"
Jul 16 22:36:28.287608 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.287426    2566 scope.go:117] "RemoveContainer" containerID="d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4"
Jul 16 22:36:28.287784 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:28.287668    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx_odh-ai-gateway-infra(712a36cd-8af7-43bd-8c8b-14619d1d0e0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b"
Jul 16 22:36:28.619289 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.619194    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:28.624053 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.624036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:28.631702 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.631673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:28.747384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.747349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd\" (UID: \"5e3118f7-c737-4f1f-b8d1-c7b493802f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:28.848208 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.848172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd\" (UID: \"5e3118f7-c737-4f1f-b8d1-c7b493802f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:28.848801 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.848777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd\" (UID: \"5e3118f7-c737-4f1f-b8d1-c7b493802f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:28.936934 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:28.936850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:29.072933 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:29.072897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:29.074490 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:36:29.074434    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3118f7_c737_4f1f_b8d1_c7b493802f8b.slice/crio-eb23bbbb38ba12ebea03bb01e51f58c209cd7dac0a3d22296a2818e618295037 WatchSource:0}: Error finding container eb23bbbb38ba12ebea03bb01e51f58c209cd7dac0a3d22296a2818e618295037: Status 404 returned error can't find the container with id eb23bbbb38ba12ebea03bb01e51f58c209cd7dac0a3d22296a2818e618295037
Jul 16 22:36:29.296985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:29.295993    2566 scope.go:117] "RemoveContainer" containerID="d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4"
Jul 16 22:36:29.296985 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:29.296248    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx_odh-ai-gateway-infra(712a36cd-8af7-43bd-8c8b-14619d1d0e0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b"
Jul 16 22:36:29.298351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:29.298322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerStarted","Data":"eb23bbbb38ba12ebea03bb01e51f58c209cd7dac0a3d22296a2818e618295037"}
Jul 16 22:36:30.305714 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:30.305675    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerID="c4cc3bdd0fe20df4203961ef205da9d4cae235c88688aeb22559d1aa2c6d0fac" exitCode=6
Jul 16 22:36:30.306114 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:30.305761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerDied","Data":"c4cc3bdd0fe20df4203961ef205da9d4cae235c88688aeb22559d1aa2c6d0fac"}
Jul 16 22:36:30.306114 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:30.305913    2566 scope.go:117] "RemoveContainer" containerID="c4cc3bdd0fe20df4203961ef205da9d4cae235c88688aeb22559d1aa2c6d0fac"
Jul 16 22:36:31.313107 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:31.313072    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerID="506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5" exitCode=6
Jul 16 22:36:31.313565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:31.313130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerDied","Data":"506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5"}
Jul 16 22:36:31.313565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:31.313227    2566 scope.go:117] "RemoveContainer" containerID="c4cc3bdd0fe20df4203961ef205da9d4cae235c88688aeb22559d1aa2c6d0fac"
Jul 16 22:36:31.313565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:31.313380    2566 scope.go:117] "RemoveContainer" containerID="506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5"
Jul 16 22:36:31.313698 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:31.313672    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd_odh-ai-gateway-infra(5e3118f7-c737-4f1f-b8d1-c7b493802f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b"
Jul 16 22:36:32.319895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:32.319857    2566 scope.go:117] "RemoveContainer" containerID="506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5"
Jul 16 22:36:32.320264 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:32.320039    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd_odh-ai-gateway-infra(5e3118f7-c737-4f1f-b8d1-c7b493802f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b"
Jul 16 22:36:41.004889 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.004855    2566 scope.go:117] "RemoveContainer" containerID="d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4"
Jul 16 22:36:41.370704 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.370615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerStarted","Data":"79f1c0daa8d9900a089fcb489187becf25981ca78338e8c19dfd5a5e51ac8dba"}
Jul 16 22:36:41.395726 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.395677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" podStartSLOduration=16.395663417 podStartE2EDuration="16.395663417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:41.394847365 +0000 UTC m=+2959.904714448" watchObservedRunningTime="2026-07-16 22:36:41.395663417 +0000 UTC m=+2959.905530499"
Jul 16 22:36:41.830532 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.830500    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:36:41.834166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.834143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:41.837183 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.837155    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8247b5-serving-cert\""
Jul 16 22:36:41.840257 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.840221    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:36:41.867790 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.867755    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:36:41.973026 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.972931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:41.973026 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.972975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:41.973221 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:41.973105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fshb\" (UniqueName: \"kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.074565 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.074527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.074988 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.074572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.074988 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.074694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fshb\" (UniqueName: \"kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.075358 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.075336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.077223 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.077196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.088618 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.088598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fshb\" (UniqueName: \"kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb\") pod \"maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.145513 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.145480    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:42.378634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.378598    2566 generic.go:356] "Generic (PLEG): container finished" podID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerID="79f1c0daa8d9900a089fcb489187becf25981ca78338e8c19dfd5a5e51ac8dba" exitCode=6
Jul 16 22:36:42.378795 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.378667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerDied","Data":"79f1c0daa8d9900a089fcb489187becf25981ca78338e8c19dfd5a5e51ac8dba"}
Jul 16 22:36:42.378795 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.378725    2566 scope.go:117] "RemoveContainer" containerID="d8d555ab16e440af708f0b68b2bbc683db597d09b9f57f211054b2cf48fd53d4"
Jul 16 22:36:42.379033 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.379019    2566 scope.go:117] "RemoveContainer" containerID="79f1c0daa8d9900a089fcb489187becf25981ca78338e8c19dfd5a5e51ac8dba"
Jul 16 22:36:42.379262 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:42.379240    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx_odh-ai-gateway-infra(712a36cd-8af7-43bd-8c8b-14619d1d0e0b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b"
Jul 16 22:36:42.425295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.425266    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:42.521727 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.521700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:36:42.523859 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:36:42.523832    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd279204e_bcce_43a8_9b6f_3292309e0a05.slice/crio-1baae23335b883b310ad1beda7a90da24235e6ff932ea545aa3885edb1ed3307 WatchSource:0}: Error finding container 1baae23335b883b310ad1beda7a90da24235e6ff932ea545aa3885edb1ed3307: Status 404 returned error can't find the container with id 1baae23335b883b310ad1beda7a90da24235e6ff932ea545aa3885edb1ed3307
Jul 16 22:36:42.525643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:42.525623    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:36:43.385421 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.385385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" event={"ID":"d279204e-bcce-43a8-9b6f-3292309e0a05","Type":"ContainerStarted","Data":"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"}
Jul 16 22:36:43.385885 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.385440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" event={"ID":"d279204e-bcce-43a8-9b6f-3292309e0a05","Type":"ContainerStarted","Data":"1baae23335b883b310ad1beda7a90da24235e6ff932ea545aa3885edb1ed3307"}
Jul 16 22:36:43.385885 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.385514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:43.433669 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.433608    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" podStartSLOduration=1.821553324 podStartE2EDuration="2.43356201s" totalImagesPullingTime="612.008686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:43.432344732 +0000 UTC m=+2961.942211814" watchObservedRunningTime="2026-07-16 22:36:43.43356201 +0000 UTC m=+2961.943429092"
Jul 16 22:36:43.518242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.518215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:43.691290 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.691216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca\") pod \"712a36cd-8af7-43bd-8c8b-14619d1d0e0b\" (UID: \"712a36cd-8af7-43bd-8c8b-14619d1d0e0b\") "
Jul 16 22:36:43.691606 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.691582    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "712a36cd-8af7-43bd-8c8b-14619d1d0e0b" (UID: "712a36cd-8af7-43bd-8c8b-14619d1d0e0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:43.792403 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:43.792377    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712a36cd-8af7-43bd-8c8b-14619d1d0e0b-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:44.394287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:44.394211    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"
Jul 16 22:36:44.394287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:44.394247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx" event={"ID":"712a36cd-8af7-43bd-8c8b-14619d1d0e0b","Type":"ContainerDied","Data":"75940c6de5f0b7469c382d43d766016883ce5b73ce4b90f80a78f4ececfb3b27"}
Jul 16 22:36:44.394774 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:44.394318    2566 scope.go:117] "RemoveContainer" containerID="79f1c0daa8d9900a089fcb489187becf25981ca78338e8c19dfd5a5e51ac8dba"
Jul 16 22:36:44.435587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:44.435553    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:44.437757 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:44.437735    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-g9jkx"]
Jul 16 22:36:45.004638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:45.004606    2566 scope.go:117] "RemoveContainer" containerID="506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5"
Jul 16 22:36:45.402744 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:45.402714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerStarted","Data":"4e1c46a4ce72cafdce9570c632bb17f957156c8e50cec1e002dc7e893e8b1719"}
Jul 16 22:36:45.429968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:45.429917    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" podStartSLOduration=17.429900978 podStartE2EDuration="17.429900978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:45.427685129 +0000 UTC m=+2963.937552211" watchObservedRunningTime="2026-07-16 22:36:45.429900978 +0000 UTC m=+2963.939768060"
Jul 16 22:36:46.011591 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.011555    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" path="/var/lib/kubelet/pods/712a36cd-8af7-43bd-8c8b-14619d1d0e0b/volumes"
Jul 16 22:36:46.410216 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.410139    2566 generic.go:356] "Generic (PLEG): container finished" podID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerID="4e1c46a4ce72cafdce9570c632bb17f957156c8e50cec1e002dc7e893e8b1719" exitCode=6
Jul 16 22:36:46.410628 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.410212    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerDied","Data":"4e1c46a4ce72cafdce9570c632bb17f957156c8e50cec1e002dc7e893e8b1719"}
Jul 16 22:36:46.410628 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.410274    2566 scope.go:117] "RemoveContainer" containerID="506d1a4bcaee0d67d4d7ab7da087724ec5e202f77700e00d896d786fe13019f5"
Jul 16 22:36:46.410735 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.410666    2566 scope.go:117] "RemoveContainer" containerID="4e1c46a4ce72cafdce9570c632bb17f957156c8e50cec1e002dc7e893e8b1719"
Jul 16 22:36:46.410889 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:36:46.410867    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd_odh-ai-gateway-infra(5e3118f7-c737-4f1f-b8d1-c7b493802f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b"
Jul 16 22:36:46.453280 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:46.453251    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:47.571299 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:47.571274    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:47.625964 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:47.625934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca\") pod \"5e3118f7-c737-4f1f-b8d1-c7b493802f8b\" (UID: \"5e3118f7-c737-4f1f-b8d1-c7b493802f8b\") "
Jul 16 22:36:47.626387 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:47.626359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e3118f7-c737-4f1f-b8d1-c7b493802f8b" (UID: "5e3118f7-c737-4f1f-b8d1-c7b493802f8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:47.726956 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:47.726877    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e3118f7-c737-4f1f-b8d1-c7b493802f8b-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:36:48.432074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:48.432000    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"
Jul 16 22:36:48.432074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:48.432041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd" event={"ID":"5e3118f7-c737-4f1f-b8d1-c7b493802f8b","Type":"ContainerDied","Data":"eb23bbbb38ba12ebea03bb01e51f58c209cd7dac0a3d22296a2818e618295037"}
Jul 16 22:36:48.432286 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:48.432102    2566 scope.go:117] "RemoveContainer" containerID="4e1c46a4ce72cafdce9570c632bb17f957156c8e50cec1e002dc7e893e8b1719"
Jul 16 22:36:48.465242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:48.465213    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:48.471560 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:48.471536    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-2k6vd"]
Jul 16 22:36:49.400304 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.400271    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:36:49.777505 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.777449    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:36:49.778217 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778194    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778222    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778253    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.778281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778262    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.778281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778275    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778313    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778322    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778344    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.778414 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778352    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.778590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778476    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.778590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778491    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778505    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778515    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="712a36cd-8af7-43bd-8c8b-14619d1d0e0b" containerName="revoke-keys"
Jul 16 22:36:49.778590 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.778525    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:36:49.783670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.783647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.786797 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.786778    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1f2f96-serving-cert\""
Jul 16 22:36:49.794420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.794389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:36:49.844182 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.844156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.844301 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.844196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9267z\" (UniqueName: \"kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.844301 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.844294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.944804 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.944767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.944958 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.944825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.944958 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.944874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9267z\" (UniqueName: \"kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.945614 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.945593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.947326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.947306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:49.955825 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:49.955801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9267z\" (UniqueName: \"kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z\") pod \"maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:50.011226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:50.011192    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" path="/var/lib/kubelet/pods/5e3118f7-c737-4f1f-b8d1-c7b493802f8b/volumes"
Jul 16 22:36:50.096304 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:50.096223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:50.246261 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:50.246219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:36:50.249040 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:36:50.249011    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cb9cc8_f78d_4919_9fcf_a09f16b6e65f.slice/crio-8af6fd90ee6514eda35823cef95e5e0ddeec18755c870b17f8461baef8f9a7a8 WatchSource:0}: Error finding container 8af6fd90ee6514eda35823cef95e5e0ddeec18755c870b17f8461baef8f9a7a8: Status 404 returned error can't find the container with id 8af6fd90ee6514eda35823cef95e5e0ddeec18755c870b17f8461baef8f9a7a8
Jul 16 22:36:50.445983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:50.445896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" event={"ID":"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f","Type":"ContainerStarted","Data":"8af6fd90ee6514eda35823cef95e5e0ddeec18755c870b17f8461baef8f9a7a8"}
Jul 16 22:36:51.454626 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:51.454576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" event={"ID":"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f","Type":"ContainerStarted","Data":"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"}
Jul 16 22:36:51.455012 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:51.454660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:57.466903 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:57.466866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:36:57.490667 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:36:57.490607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" podStartSLOduration=8.14695407 podStartE2EDuration="8.490590037s" totalImagesPullingTime="343.635967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:51.49152275 +0000 UTC m=+2970.001389834" watchObservedRunningTime="2026-07-16 22:36:57.490590037 +0000 UTC m=+2976.000457119"
Jul 16 22:37:12.169903 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.169861    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:37:12.170380 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.170206    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" podUID="d279204e-bcce-43a8-9b6f-3292309e0a05" containerName="maas-api" containerID="cri-o://af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478" gracePeriod=30
Jul 16 22:37:12.434725 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.434658    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:37:12.556991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.556945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fshb\" (UniqueName: \"kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb\") pod \"d279204e-bcce-43a8-9b6f-3292309e0a05\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") "
Jul 16 22:37:12.556991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.556998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls\") pod \"d279204e-bcce-43a8-9b6f-3292309e0a05\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") "
Jul 16 22:37:12.557254 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.557102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca\") pod \"d279204e-bcce-43a8-9b6f-3292309e0a05\" (UID: \"d279204e-bcce-43a8-9b6f-3292309e0a05\") "
Jul 16 22:37:12.557505 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.557475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d279204e-bcce-43a8-9b6f-3292309e0a05" (UID: "d279204e-bcce-43a8-9b6f-3292309e0a05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:12.559215 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.559191    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d279204e-bcce-43a8-9b6f-3292309e0a05" (UID: "d279204e-bcce-43a8-9b6f-3292309e0a05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:37:12.559309 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.559279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb" (OuterVolumeSpecName: "kube-api-access-8fshb") pod "d279204e-bcce-43a8-9b6f-3292309e0a05" (UID: "d279204e-bcce-43a8-9b6f-3292309e0a05"). InnerVolumeSpecName "kube-api-access-8fshb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:37:12.574682 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.574651    2566 generic.go:356] "Generic (PLEG): container finished" podID="d279204e-bcce-43a8-9b6f-3292309e0a05" containerID="af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478" exitCode=0
Jul 16 22:37:12.574805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.574696    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"
Jul 16 22:37:12.574805 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.574688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" event={"ID":"d279204e-bcce-43a8-9b6f-3292309e0a05","Type":"ContainerDied","Data":"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"}
Jul 16 22:37:12.574882 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.574810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9" event={"ID":"d279204e-bcce-43a8-9b6f-3292309e0a05","Type":"ContainerDied","Data":"1baae23335b883b310ad1beda7a90da24235e6ff932ea545aa3885edb1ed3307"}
Jul 16 22:37:12.574882 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.574838    2566 scope.go:117] "RemoveContainer" containerID="af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"
Jul 16 22:37:12.584345 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.584318    2566 scope.go:117] "RemoveContainer" containerID="af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"
Jul 16 22:37:12.584652 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:37:12.584629    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478\": container with ID starting with af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478 not found: ID does not exist" containerID="af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"
Jul 16 22:37:12.584739 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.584668    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478"} err="failed to get container status \"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478\": rpc error: code = NotFound desc = could not find container \"af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478\": container with ID starting with af539caccd14530a67ac0288e8d1448cf2402ffa7bb225e49246c5d744adf478 not found: ID does not exist"
Jul 16 22:37:12.597235 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.597201    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:37:12.603315 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.603287    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8247b5-6df87d6dc5-v6st9"]
Jul 16 22:37:12.658752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.658715    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fshb\" (UniqueName: \"kubernetes.io/projected/d279204e-bcce-43a8-9b6f-3292309e0a05-kube-api-access-8fshb\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:12.658752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.658746    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d279204e-bcce-43a8-9b6f-3292309e0a05-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:12.658752 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:12.658757    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d279204e-bcce-43a8-9b6f-3292309e0a05-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:14.011807 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:14.011765    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d279204e-bcce-43a8-9b6f-3292309e0a05" path="/var/lib/kubelet/pods/d279204e-bcce-43a8-9b6f-3292309e0a05/volumes"
Jul 16 22:37:18.679146 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679106    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:18.679619 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679605    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d279204e-bcce-43a8-9b6f-3292309e0a05" containerName="maas-api"
Jul 16 22:37:18.679619 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679620    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d279204e-bcce-43a8-9b6f-3292309e0a05" containerName="maas-api"
Jul 16 22:37:18.679704 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679633    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:37:18.679704 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679639    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:37:18.679771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679710    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d279204e-bcce-43a8-9b6f-3292309e0a05" containerName="maas-api"
Jul 16 22:37:18.679771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.679720    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3118f7-c737-4f1f-b8d1-c7b493802f8b" containerName="revoke-keys"
Jul 16 22:37:18.684514 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.684493    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:18.687913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.687888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:37:18.690911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.690882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:18.822297 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.822239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5\" (UID: \"80d734a7-b28e-4ed7-a120-7f29063183d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:18.923447 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.923405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5\" (UID: \"80d734a7-b28e-4ed7-a120-7f29063183d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:18.924076 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.924056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5\" (UID: \"80d734a7-b28e-4ed7-a120-7f29063183d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:18.997646 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:18.997615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:19.147341 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:19.147315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:19.149545 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:37:19.149513    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d734a7_b28e_4ed7_a120_7f29063183d1.slice/crio-2358037b4760353ddb82d1e05b00097349b151122cf01d14710d97f42864c973 WatchSource:0}: Error finding container 2358037b4760353ddb82d1e05b00097349b151122cf01d14710d97f42864c973: Status 404 returned error can't find the container with id 2358037b4760353ddb82d1e05b00097349b151122cf01d14710d97f42864c973
Jul 16 22:37:19.619147 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:19.619050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerStarted","Data":"40d8454eebc4738781edbb9e6a1fd0edab73d1c36a4dd89f6130ecb29d406401"}
Jul 16 22:37:19.619147 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:19.619107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerStarted","Data":"2358037b4760353ddb82d1e05b00097349b151122cf01d14710d97f42864c973"}
Jul 16 22:37:19.636373 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:19.636313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" podStartSLOduration=1.636295144 podStartE2EDuration="1.636295144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:19.634743975 +0000 UTC m=+2998.144611056" watchObservedRunningTime="2026-07-16 22:37:19.636295144 +0000 UTC m=+2998.146162227"
Jul 16 22:37:20.626766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:20.626728    2566 generic.go:356] "Generic (PLEG): container finished" podID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerID="40d8454eebc4738781edbb9e6a1fd0edab73d1c36a4dd89f6130ecb29d406401" exitCode=6
Jul 16 22:37:20.627176 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:20.626809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerDied","Data":"40d8454eebc4738781edbb9e6a1fd0edab73d1c36a4dd89f6130ecb29d406401"}
Jul 16 22:37:20.627176 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:20.627024    2566 scope.go:117] "RemoveContainer" containerID="40d8454eebc4738781edbb9e6a1fd0edab73d1c36a4dd89f6130ecb29d406401"
Jul 16 22:37:21.635030 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:21.634937    2566 generic.go:356] "Generic (PLEG): container finished" podID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerID="020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8" exitCode=6
Jul 16 22:37:21.635030 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:21.635015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerDied","Data":"020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8"}
Jul 16 22:37:21.635482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:21.635081    2566 scope.go:117] "RemoveContainer" containerID="40d8454eebc4738781edbb9e6a1fd0edab73d1c36a4dd89f6130ecb29d406401"
Jul 16 22:37:21.635482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:21.635323    2566 scope.go:117] "RemoveContainer" containerID="020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8"
Jul 16 22:37:21.635592 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:37:21.635572    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5_odh-ai-gateway-infra(80d734a7-b28e-4ed7-a120-7f29063183d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1"
Jul 16 22:37:22.214137 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:22.214107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:37:22.222148 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:22.222122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:37:22.642801 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:22.642770    2566 scope.go:117] "RemoveContainer" containerID="020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8"
Jul 16 22:37:22.643169 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:37:22.642999    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5_odh-ai-gateway-infra(80d734a7-b28e-4ed7-a120-7f29063183d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1"
Jul 16 22:37:34.005293 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:34.005253    2566 scope.go:117] "RemoveContainer" containerID="020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8"
Jul 16 22:37:34.711334 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:34.711297    2566 generic.go:356] "Generic (PLEG): container finished" podID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerID="db0b373038e695c1a10a79fa06ba036cc03712dc5198f5f7e7f7bc0587f6f7ff" exitCode=6
Jul 16 22:37:34.711543 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:34.711370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerDied","Data":"db0b373038e695c1a10a79fa06ba036cc03712dc5198f5f7e7f7bc0587f6f7ff"}
Jul 16 22:37:34.711543 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:34.711436    2566 scope.go:117] "RemoveContainer" containerID="020d8f2b979e808e0c551c6baca6422d8c7676bdd33c11fc3718eed9bbeb42e8"
Jul 16 22:37:34.711784 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:34.711761    2566 scope.go:117] "RemoveContainer" containerID="db0b373038e695c1a10a79fa06ba036cc03712dc5198f5f7e7f7bc0587f6f7ff"
Jul 16 22:37:34.712073 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:37:34.712052    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5_odh-ai-gateway-infra(80d734a7-b28e-4ed7-a120-7f29063183d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1"
Jul 16 22:37:35.746516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:35.746474    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:35.887284 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:35.887259    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:35.985986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:35.985956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca\") pod \"80d734a7-b28e-4ed7-a120-7f29063183d1\" (UID: \"80d734a7-b28e-4ed7-a120-7f29063183d1\") "
Jul 16 22:37:35.986382 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:35.986354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80d734a7-b28e-4ed7-a120-7f29063183d1" (UID: "80d734a7-b28e-4ed7-a120-7f29063183d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:36.087240 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.087188    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80d734a7-b28e-4ed7-a120-7f29063183d1-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:36.726683 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.726648    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"
Jul 16 22:37:36.726897 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.726687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5" event={"ID":"80d734a7-b28e-4ed7-a120-7f29063183d1","Type":"ContainerDied","Data":"2358037b4760353ddb82d1e05b00097349b151122cf01d14710d97f42864c973"}
Jul 16 22:37:36.726897 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.726762    2566 scope.go:117] "RemoveContainer" containerID="db0b373038e695c1a10a79fa06ba036cc03712dc5198f5f7e7f7bc0587f6f7ff"
Jul 16 22:37:36.746326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.746279    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:36.748542 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:36.748518    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-s7cg5"]
Jul 16 22:37:38.011962 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:38.011924    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" path="/var/lib/kubelet/pods/80d734a7-b28e-4ed7-a120-7f29063183d1/volumes"
Jul 16 22:37:38.765753 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:38.765707    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:37:38.766242 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:38.766210    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" podUID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" containerName="maas-api" containerID="cri-o://389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4" gracePeriod=30
Jul 16 22:37:39.047171 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.047143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:37:39.116187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.116145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls\") pod \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") "
Jul 16 22:37:39.116187 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.116195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca\") pod \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") "
Jul 16 22:37:39.116487 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.116279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9267z\" (UniqueName: \"kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z\") pod \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\" (UID: \"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f\") "
Jul 16 22:37:39.116751 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.116716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" (UID: "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:39.118554 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.118513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z" (OuterVolumeSpecName: "kube-api-access-9267z") pod "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" (UID: "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f"). InnerVolumeSpecName "kube-api-access-9267z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:37:39.118652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.118549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" (UID: "a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:37:39.217737 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.217691    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:39.217737 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.217729    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:39.217737 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.217740    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9267z\" (UniqueName: \"kubernetes.io/projected/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f-kube-api-access-9267z\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:37:39.747586 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.747542    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" containerID="389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4" exitCode=0
Jul 16 22:37:39.747793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.747590    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"
Jul 16 22:37:39.747793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.747591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" event={"ID":"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f","Type":"ContainerDied","Data":"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"}
Jul 16 22:37:39.747793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.747735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9" event={"ID":"a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f","Type":"ContainerDied","Data":"8af6fd90ee6514eda35823cef95e5e0ddeec18755c870b17f8461baef8f9a7a8"}
Jul 16 22:37:39.747793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.747759    2566 scope.go:117] "RemoveContainer" containerID="389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"
Jul 16 22:37:39.758023 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.758002    2566 scope.go:117] "RemoveContainer" containerID="389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"
Jul 16 22:37:39.758343 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:37:39.758324    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4\": container with ID starting with 389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4 not found: ID does not exist" containerID="389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"
Jul 16 22:37:39.758398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.758352    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4"} err="failed to get container status \"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4\": rpc error: code = NotFound desc = could not find container \"389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4\": container with ID starting with 389e0eb23d7c6b1c79c44ae0ada50d29e37582346682be2c367df242a8f0c0b4 not found: ID does not exist"
Jul 16 22:37:39.784009 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.783968    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:37:39.792263 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:39.792230    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f2f96-85c7d889d-mljd9"]
Jul 16 22:37:40.011014 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:40.010926    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" path="/var/lib/kubelet/pods/a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f/volumes"
Jul 16 22:37:58.036400 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036362    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036831    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036843    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036851    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036871    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036881    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" containerName="maas-api"
Jul 16 22:37:58.036913 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036887    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" containerName="maas-api"
Jul 16 22:37:58.037340 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036951    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2cb9cc8-f78d-4919-9fcf-a09f16b6e65f" containerName="maas-api"
Jul 16 22:37:58.037340 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036960    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.037340 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.036966    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:37:58.041587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.041564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.046114 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.046092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hs625\""
Jul 16 22:37:58.046637 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.046611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1c9596-serving-cert\""
Jul 16 22:37:58.055898 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.055872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:37:58.209284 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.209245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.209503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.209347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnk7d\" (UniqueName: \"kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.209503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.209440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.310992 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.310864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.311173 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.310991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.311173 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.311129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnk7d\" (UniqueName: \"kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.313297 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.313261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.319368 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.319339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.325476 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.325441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnk7d\" (UniqueName: \"kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d\") pod \"maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.354580 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.354556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:58.506334 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.506308    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:37:58.508507 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:37:58.508474    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a4deaa_1365_4bbd_89fc_dd05b3bfa6bd.slice/crio-c36087de963b6805d7269fb65de9ef856feaf988a8e70878eb6ad1bfd612ef60 WatchSource:0}: Error finding container c36087de963b6805d7269fb65de9ef856feaf988a8e70878eb6ad1bfd612ef60: Status 404 returned error can't find the container with id c36087de963b6805d7269fb65de9ef856feaf988a8e70878eb6ad1bfd612ef60
Jul 16 22:37:58.859210 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:58.859168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" event={"ID":"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd","Type":"ContainerStarted","Data":"c36087de963b6805d7269fb65de9ef856feaf988a8e70878eb6ad1bfd612ef60"}
Jul 16 22:37:59.866658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:59.866615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" event={"ID":"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd","Type":"ContainerStarted","Data":"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"}
Jul 16 22:37:59.867118 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:59.866685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:37:59.885652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:37:59.885594    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" podStartSLOduration=1.35291011 podStartE2EDuration="1.885575443s" totalImagesPullingTime="532.665333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:59.883508 +0000 UTC m=+3038.393375076" watchObservedRunningTime="2026-07-16 22:37:59.885575443 +0000 UTC m=+3038.395442524"
Jul 16 22:38:05.878559 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:05.878522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:38:06.124608 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.124566    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:38:06.125256 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.125237    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d734a7-b28e-4ed7-a120-7f29063183d1" containerName="revoke-keys"
Jul 16 22:38:06.128634 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.128567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.132319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.132290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-7d2f73-serving-cert\""
Jul 16 22:38:06.149397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.149358    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:38:06.192508 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.192437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlbvg\" (UniqueName: \"kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.192693 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.192575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.192693 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.192610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.293303 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.293264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.293303 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.293304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.293596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.293366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlbvg\" (UniqueName: \"kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.294040 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.294014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.295896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.295874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.308120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.308096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlbvg\" (UniqueName: \"kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg\") pod \"maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.441385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.441289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:06.592188 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.592159    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:38:06.593884 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:38:06.593855    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bff337d_99eb_45f0_94a3_9c9dc5b87e0a.slice/crio-bd4576579f3884d1cbe069ee3da447b3351b187ce558df9fa0ef7c44832fca74 WatchSource:0}: Error finding container bd4576579f3884d1cbe069ee3da447b3351b187ce558df9fa0ef7c44832fca74: Status 404 returned error can't find the container with id bd4576579f3884d1cbe069ee3da447b3351b187ce558df9fa0ef7c44832fca74
Jul 16 22:38:06.906256 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:06.906220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" event={"ID":"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a","Type":"ContainerStarted","Data":"bd4576579f3884d1cbe069ee3da447b3351b187ce558df9fa0ef7c44832fca74"}
Jul 16 22:38:07.914690 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:07.914655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" event={"ID":"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a","Type":"ContainerStarted","Data":"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"}
Jul 16 22:38:07.915134 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:07.914773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:07.939933 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:07.939879    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" podStartSLOduration=1.6381227200000001 podStartE2EDuration="1.939859473s" totalImagesPullingTime="301.736753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:07.939206667 +0000 UTC m=+3046.449073749" watchObservedRunningTime="2026-07-16 22:38:07.939859473 +0000 UTC m=+3046.449726554"
Jul 16 22:38:13.926197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:13.926160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:38:27.398536 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.398502    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:38:27.403830 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.403808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.409325 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.409303    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"kube-root-ca.crt\""
Jul 16 22:38:27.410359 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.410340    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"openshift-service-ca.crt\""
Jul 16 22:38:27.410468 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.410353    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"auth-test-model-1c9596-kserve-self-signed-certs\""
Jul 16 22:38:27.410468 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.410357    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"default-dockercfg-wxbsp\""
Jul 16 22:38:27.416180 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.416160    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:38:27.485185 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.485369 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq2p\" (UniqueName: \"kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.485432 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.485505 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.485562 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.485602 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.485568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586773 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.586978 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.586889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq2p\" (UniqueName: \"kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.587282 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.587229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.587282 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.587250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.587359 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.587284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.589233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.589208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.589416 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.589399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.595611 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.595585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq2p\" (UniqueName: \"kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p\") pod \"auth-test-model-1c9596-kserve-bb695586c-tglwz\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") " pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.716335 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.716255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:27.883793 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:27.883752    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:38:27.894124 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:38:27.894089    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0060665c_a2a0_460f_aef5_8384fc1666b6.slice/crio-6c019290f66149b00690fadf34161b6f5ff78ba6fc22c453dae7232c3460332e WatchSource:0}: Error finding container 6c019290f66149b00690fadf34161b6f5ff78ba6fc22c453dae7232c3460332e: Status 404 returned error can't find the container with id 6c019290f66149b00690fadf34161b6f5ff78ba6fc22c453dae7232c3460332e
Jul 16 22:38:28.026778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:28.026740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" event={"ID":"0060665c-a2a0-460f-aef5-8384fc1666b6","Type":"ContainerStarted","Data":"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"}
Jul 16 22:38:28.026778 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:28.026790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" event={"ID":"0060665c-a2a0-460f-aef5-8384fc1666b6","Type":"ContainerStarted","Data":"6c019290f66149b00690fadf34161b6f5ff78ba6fc22c453dae7232c3460332e"}
Jul 16 22:38:28.059849 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:28.059789    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" podStartSLOduration=1.059774446 podStartE2EDuration="1.059774446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:28.056412186 +0000 UTC m=+3066.566279268" watchObservedRunningTime="2026-07-16 22:38:28.059774446 +0000 UTC m=+3066.569641528"
Jul 16 22:38:37.717120 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:37.717084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:37.717558 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:37.717133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:37.730032 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:37.730009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:38:38.090721 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:38:38.090691    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:40:52.240967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:40:52.240934    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:41:07.261032 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.260968    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" podUID="0060665c-a2a0-460f-aef5-8384fc1666b6" containerName="main" containerID="cri-o://6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c" gracePeriod=45
Jul 16 22:41:07.507407 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.507384    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:41:07.588716 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588650    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgq2p\" (UniqueName: \"kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.588716 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.588920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.588920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.588920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588842    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.588920 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.588895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache\") pod \"0060665c-a2a0-460f-aef5-8384fc1666b6\" (UID: \"0060665c-a2a0-460f-aef5-8384fc1666b6\") "
Jul 16 22:41:07.589131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.589078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home" (OuterVolumeSpecName: "home") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:07.589264 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.589247    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-home\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.589356 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.589325    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:07.589398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.589343    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:07.590983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.590947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm" (OuterVolumeSpecName: "dshm") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:07.591075 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.590996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:41:07.591075 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.591035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p" (OuterVolumeSpecName: "kube-api-access-qgq2p") pod "0060665c-a2a0-460f-aef5-8384fc1666b6" (UID: "0060665c-a2a0-460f-aef5-8384fc1666b6"). InnerVolumeSpecName "kube-api-access-qgq2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:07.690719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.690688    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-dshm\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.690719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.690713    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-model-cache\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.690719 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.690725    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq2p\" (UniqueName: \"kubernetes.io/projected/0060665c-a2a0-460f-aef5-8384fc1666b6-kube-api-access-qgq2p\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.690917 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.690734    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0060665c-a2a0-460f-aef5-8384fc1666b6-tmp-dir\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.690917 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.690742    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0060665c-a2a0-460f-aef5-8384fc1666b6-tls-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:41:07.882997 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.882911    2566 generic.go:356] "Generic (PLEG): container finished" podID="0060665c-a2a0-460f-aef5-8384fc1666b6" containerID="6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c" exitCode=0
Jul 16 22:41:07.882997 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.882963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"
Jul 16 22:41:07.883165 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.882997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" event={"ID":"0060665c-a2a0-460f-aef5-8384fc1666b6","Type":"ContainerDied","Data":"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"}
Jul 16 22:41:07.883165 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.883045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz" event={"ID":"0060665c-a2a0-460f-aef5-8384fc1666b6","Type":"ContainerDied","Data":"6c019290f66149b00690fadf34161b6f5ff78ba6fc22c453dae7232c3460332e"}
Jul 16 22:41:07.883165 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.883064    2566 scope.go:117] "RemoveContainer" containerID="6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"
Jul 16 22:41:07.892158 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.892140    2566 scope.go:117] "RemoveContainer" containerID="6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"
Jul 16 22:41:07.892414 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:41:07.892396    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c\": container with ID starting with 6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c not found: ID does not exist" containerID="6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"
Jul 16 22:41:07.892502 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.892422    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c"} err="failed to get container status \"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c\": rpc error: code = NotFound desc = could not find container \"6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c\": container with ID starting with 6828b2b46a608b669afc8b6d53a7b034c12d9b104d19b028e2938b286673648c not found: ID does not exist"
Jul 16 22:41:07.904627 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.904605    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:41:07.910824 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:07.910804    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/auth-test-model-1c9596-kserve-bb695586c-tglwz"]
Jul 16 22:41:08.012253 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:08.012221    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0060665c-a2a0-460f-aef5-8384fc1666b6" path="/var/lib/kubelet/pods/0060665c-a2a0-460f-aef5-8384fc1666b6/volumes"
Jul 16 22:41:14.394522 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.394435    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:41:14.394931 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.394916    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0060665c-a2a0-460f-aef5-8384fc1666b6" containerName="main"
Jul 16 22:41:14.394974 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.394934    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0060665c-a2a0-460f-aef5-8384fc1666b6" containerName="main"
Jul 16 22:41:14.395035 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.395025    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0060665c-a2a0-460f-aef5-8384fc1666b6" containerName="main"
Jul 16 22:41:14.399430 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.399411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.402939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.402918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"sub-test-model-1c9596-kserve-self-signed-certs\""
Jul 16 22:41:14.402939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.402931    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"openshift-service-ca.crt\""
Jul 16 22:41:14.403096 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.402924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"kube-root-ca.crt\""
Jul 16 22:41:14.403096 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.402936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1c9596\"/\"default-dockercfg-wxbsp\""
Jul 16 22:41:14.410972 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.410942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:41:14.454068 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.454206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.454206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.454206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgz22\" (UniqueName: \"kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.454206 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.454356 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.454207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555566 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgz22\" (UniqueName: \"kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555728 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.555928 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.555783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.556216 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.556188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.556342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.556250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.556342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.556295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.557895 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.557863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.558181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.558162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.565087 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.565060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgz22\" (UniqueName: \"kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22\") pod \"sub-test-model-1c9596-kserve-6f597b9d45-ffcq4\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") " pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.709936 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.709836    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:14.846985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.846956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:41:14.848676 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:41:14.848651    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd8bb9e_00fe_4991_80bb_0ae4e4c62867.slice/crio-6aeb604fffb3f521c899748308ffdf9a679a8dbc8f9016f09ccbc69f6042d652 WatchSource:0}: Error finding container 6aeb604fffb3f521c899748308ffdf9a679a8dbc8f9016f09ccbc69f6042d652: Status 404 returned error can't find the container with id 6aeb604fffb3f521c899748308ffdf9a679a8dbc8f9016f09ccbc69f6042d652
Jul 16 22:41:14.924562 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.924530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" event={"ID":"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867","Type":"ContainerStarted","Data":"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"}
Jul 16 22:41:14.924691 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.924576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" event={"ID":"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867","Type":"ContainerStarted","Data":"6aeb604fffb3f521c899748308ffdf9a679a8dbc8f9016f09ccbc69f6042d652"}
Jul 16 22:41:14.944569 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:14.944507    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" podStartSLOduration=0.94449001 podStartE2EDuration="944.49001ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:14.94331441 +0000 UTC m=+3233.453181490" watchObservedRunningTime="2026-07-16 22:41:14.94449001 +0000 UTC m=+3233.454357087"
Jul 16 22:41:24.710480 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:24.710406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:24.710938 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:24.710498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:24.722622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:24.722594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:41:24.996287 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:41:24.996260    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:42:20.554394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:20.554362    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:42:22.248880 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:22.248848    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:42:22.259567 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:22.259546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:42:35.574337 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.574271    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" podUID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" containerName="main" containerID="cri-o://e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480" gracePeriod=45
Jul 16 22:42:35.825934 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.825873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:42:35.888234 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888207    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888254    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgz22\" (UniqueName: \"kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22\") pod \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\" (UID: \"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867\") "
Jul 16 22:42:35.888688 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home" (OuterVolumeSpecName: "home") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:35.888688 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888576    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:35.888688 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:35.888827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888748    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-home\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:35.888827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888760    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-model-cache\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:35.888827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.888770    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tmp-dir\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:35.890427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.890380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:42:35.890427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.890378    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22" (OuterVolumeSpecName: "kube-api-access-rgz22") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "kube-api-access-rgz22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:42:35.890600 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.890474    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm" (OuterVolumeSpecName: "dshm") pod "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" (UID: "6cd8bb9e-00fe-4991-80bb-0ae4e4c62867"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:35.990060 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.990025    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-tls-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:35.990060 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.990057    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgz22\" (UniqueName: \"kubernetes.io/projected/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-kube-api-access-rgz22\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:35.990241 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:35.990068    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867-dshm\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:42:36.370033 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.370000    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" containerID="e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480" exitCode=0
Jul 16 22:42:36.370226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.370042    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"
Jul 16 22:42:36.370226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.370078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" event={"ID":"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867","Type":"ContainerDied","Data":"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"}
Jul 16 22:42:36.370226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.370120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4" event={"ID":"6cd8bb9e-00fe-4991-80bb-0ae4e4c62867","Type":"ContainerDied","Data":"6aeb604fffb3f521c899748308ffdf9a679a8dbc8f9016f09ccbc69f6042d652"}
Jul 16 22:42:36.370226 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.370136    2566 scope.go:117] "RemoveContainer" containerID="e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"
Jul 16 22:42:36.379091 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.379072    2566 scope.go:117] "RemoveContainer" containerID="e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"
Jul 16 22:42:36.379336 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:42:36.379319    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480\": container with ID starting with e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480 not found: ID does not exist" containerID="e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"
Jul 16 22:42:36.379379 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.379345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480"} err="failed to get container status \"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480\": rpc error: code = NotFound desc = could not find container \"e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480\": container with ID starting with e5df680d8da8f991b2b1e751a3dab67b076cbe07f34cd90f0c27561350119480 not found: ID does not exist"
Jul 16 22:42:36.392083 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.392058    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:42:36.396742 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:36.396719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1c9596/sub-test-model-1c9596-kserve-6f597b9d45-ffcq4"]
Jul 16 22:42:38.013119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:38.013078    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" path="/var/lib/kubelet/pods/6cd8bb9e-00fe-4991-80bb-0ae4e4c62867/volumes"
Jul 16 22:42:49.746110 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.746031    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:42:49.746597 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.746582    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" containerName="main"
Jul 16 22:42:49.746652 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.746599    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" containerName="main"
Jul 16 22:42:49.746686 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.746682    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd8bb9e-00fe-4991-80bb-0ae4e4c62867" containerName="main"
Jul 16 22:42:49.750951 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.750934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.755334 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.755308    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"default-dockercfg-ftw2r\""
Jul 16 22:42:49.755537 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.755309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"rate-test-model-7d2f73-kserve-self-signed-certs\""
Jul 16 22:42:49.755537 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.755364    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"openshift-service-ca.crt\""
Jul 16 22:42:49.755537 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.755315    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2f73\"/\"kube-root-ca.crt\""
Jul 16 22:42:49.778475 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.778433    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:42:49.818368 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.818540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.818540 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.818622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkq5\" (UniqueName: \"kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.818622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.818622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.818617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.919983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.919949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.919992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkq5\" (UniqueName: \"kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920489 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920730 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.920730 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.920686    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.922845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.922817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.923035 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.923019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:49.930492 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:49.930469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkq5\" (UniqueName: \"kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5\") pod \"rate-test-model-7d2f73-kserve-bfd5d486c-pzct5\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") " pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:50.060666 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:50.060574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:42:50.203383 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:50.203352    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:42:50.204878 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:42:50.204846    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87559132_785e_43c9_9b28_b85b2e26553f.slice/crio-bc577f21b4bbf6222ec92f886dc9d15ad9fbc66f681be175b9664cc09aa5dea3 WatchSource:0}: Error finding container bc577f21b4bbf6222ec92f886dc9d15ad9fbc66f681be175b9664cc09aa5dea3: Status 404 returned error can't find the container with id bc577f21b4bbf6222ec92f886dc9d15ad9fbc66f681be175b9664cc09aa5dea3
Jul 16 22:42:50.452484 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:50.452359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" event={"ID":"87559132-785e-43c9-9b28-b85b2e26553f","Type":"ContainerStarted","Data":"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"}
Jul 16 22:42:50.452484 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:50.452402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" event={"ID":"87559132-785e-43c9-9b28-b85b2e26553f","Type":"ContainerStarted","Data":"bc577f21b4bbf6222ec92f886dc9d15ad9fbc66f681be175b9664cc09aa5dea3"}
Jul 16 22:42:50.490876 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:42:50.490834    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" podStartSLOduration=1.490820232 podStartE2EDuration="1.490820232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:50.488297703 +0000 UTC m=+3328.998164784" watchObservedRunningTime="2026-07-16 22:42:50.490820232 +0000 UTC m=+3329.000687312"
Jul 16 22:43:00.060889 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:00.060861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:00.061482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:00.060908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:00.073253 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:00.073230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:00.517723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:00.517693    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:35.090917 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:35.090884    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:43:39.099500 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.099462    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852393993008"]
Jul 16 22:43:39.103332 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.103309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:39.106680 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.106660    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b2kkw\""
Jul 16 22:43:39.134099 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.134073    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852393993008"]
Jul 16 22:43:39.168571 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.168539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fzp\" (UniqueName: \"kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp\") pod \"test-curl-7472-139852393993008\" (UID: \"6b0a7d6a-053c-4360-a371-0891bc487976\") " pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:39.269806 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.269772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fzp\" (UniqueName: \"kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp\") pod \"test-curl-7472-139852393993008\" (UID: \"6b0a7d6a-053c-4360-a371-0891bc487976\") " pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:39.313523 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.313495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fzp\" (UniqueName: \"kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp\") pod \"test-curl-7472-139852393993008\" (UID: \"6b0a7d6a-053c-4360-a371-0891bc487976\") " pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:39.411657 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.411592    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:39.556235 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.556213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852393993008"]
Jul 16 22:43:39.557943 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:43:39.557919    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b0a7d6a_053c_4360_a371_0891bc487976.slice/crio-5e25aff47bcbc7077f8ce5fd34a062f1eda0031343725b36ce3686a17e804ff5 WatchSource:0}: Error finding container 5e25aff47bcbc7077f8ce5fd34a062f1eda0031343725b36ce3686a17e804ff5: Status 404 returned error can't find the container with id 5e25aff47bcbc7077f8ce5fd34a062f1eda0031343725b36ce3686a17e804ff5
Jul 16 22:43:39.559247 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.559231    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:43:39.725233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:39.725153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852393993008" event={"ID":"6b0a7d6a-053c-4360-a371-0891bc487976","Type":"ContainerStarted","Data":"5e25aff47bcbc7077f8ce5fd34a062f1eda0031343725b36ce3686a17e804ff5"}
Jul 16 22:43:40.732151 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:40.732109    2566 generic.go:356] "Generic (PLEG): container finished" podID="6b0a7d6a-053c-4360-a371-0891bc487976" containerID="70ccd28d17d906f609568318892ed4a643d5a5583d3cdc534f6f6241b6dedda7" exitCode=0
Jul 16 22:43:40.732612 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:40.732189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852393993008" event={"ID":"6b0a7d6a-053c-4360-a371-0891bc487976","Type":"ContainerDied","Data":"70ccd28d17d906f609568318892ed4a643d5a5583d3cdc534f6f6241b6dedda7"}
Jul 16 22:43:41.875344 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.875316    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:41.913301 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.913273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852393993008_6b0a7d6a-053c-4360-a371-0891bc487976/test-curl-7472-139852393993008/0.log"
Jul 16 22:43:41.937910 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.937881    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852393993008"]
Jul 16 22:43:41.941983 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.941963    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852393993008"]
Jul 16 22:43:41.995127 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.995101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4fzp\" (UniqueName: \"kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp\") pod \"6b0a7d6a-053c-4360-a371-0891bc487976\" (UID: \"6b0a7d6a-053c-4360-a371-0891bc487976\") "
Jul 16 22:43:41.997213 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:41.997185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp" (OuterVolumeSpecName: "kube-api-access-v4fzp") pod "6b0a7d6a-053c-4360-a371-0891bc487976" (UID: "6b0a7d6a-053c-4360-a371-0891bc487976"). InnerVolumeSpecName "kube-api-access-v4fzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:42.011090 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.011064    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b0a7d6a-053c-4360-a371-0891bc487976" path="/var/lib/kubelet/pods/6b0a7d6a-053c-4360-a371-0891bc487976/volumes"
Jul 16 22:43:42.096428 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.096398    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fzp\" (UniqueName: \"kubernetes.io/projected/6b0a7d6a-053c-4360-a371-0891bc487976-kube-api-access-v4fzp\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:42.119162 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.119131    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:42.119609 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.119595    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0a7d6a-053c-4360-a371-0891bc487976" containerName="test-curl-7472-139852393993008"
Jul 16 22:43:42.119665 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.119611    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0a7d6a-053c-4360-a371-0891bc487976" containerName="test-curl-7472-139852393993008"
Jul 16 22:43:42.119709 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.119700    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0a7d6a-053c-4360-a371-0891bc487976" containerName="test-curl-7472-139852393993008"
Jul 16 22:43:42.122875 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.122861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:42.143373 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.143348    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:42.197858 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.197831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzshf\" (UniqueName: \"kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf\") pod \"test-curl-7472-139852394112320\" (UID: \"7f507ea7-92ac-459a-b869-3a0dac1cfba1\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:42.298591 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.298516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzshf\" (UniqueName: \"kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf\") pod \"test-curl-7472-139852394112320\" (UID: \"7f507ea7-92ac-459a-b869-3a0dac1cfba1\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:42.309355 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.309321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzshf\" (UniqueName: \"kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf\") pod \"test-curl-7472-139852394112320\" (UID: \"7f507ea7-92ac-459a-b869-3a0dac1cfba1\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:42.431212 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.431183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:42.744510 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.744479    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852393993008"
Jul 16 22:43:42.744699 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.744509    2566 scope.go:117] "RemoveContainer" containerID="70ccd28d17d906f609568318892ed4a643d5a5583d3cdc534f6f6241b6dedda7"
Jul 16 22:43:42.792419 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:42.792399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:42.794685 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:43:42.794654    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f507ea7_92ac_459a_b869_3a0dac1cfba1.slice/crio-eac5b555474fdff6af589e32088f928ecc550f062f3ba5d4a69665ab27f7299b WatchSource:0}: Error finding container eac5b555474fdff6af589e32088f928ecc550f062f3ba5d4a69665ab27f7299b: Status 404 returned error can't find the container with id eac5b555474fdff6af589e32088f928ecc550f062f3ba5d4a69665ab27f7299b
Jul 16 22:43:43.753140 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:43.753105    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" containerID="84d35068a0e428350b00b5714ca2b5813994b30c9d7eeb7c6d000c0d66914304" exitCode=0
Jul 16 22:43:43.753513 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:43.753158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394112320" event={"ID":"7f507ea7-92ac-459a-b869-3a0dac1cfba1","Type":"ContainerDied","Data":"84d35068a0e428350b00b5714ca2b5813994b30c9d7eeb7c6d000c0d66914304"}
Jul 16 22:43:43.753513 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:43.753184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394112320" event={"ID":"7f507ea7-92ac-459a-b869-3a0dac1cfba1","Type":"ContainerStarted","Data":"eac5b555474fdff6af589e32088f928ecc550f062f3ba5d4a69665ab27f7299b"}
Jul 16 22:43:44.897574 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:44.897546    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:44.919625 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:44.919568    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852394112320_7f507ea7-92ac-459a-b869-3a0dac1cfba1/test-curl-7472-139852394112320/0.log"
Jul 16 22:43:44.943914 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:44.943886    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:44.947968 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:44.947942    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:45.023343 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.023265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzshf\" (UniqueName: \"kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf\") pod \"7f507ea7-92ac-459a-b869-3a0dac1cfba1\" (UID: \"7f507ea7-92ac-459a-b869-3a0dac1cfba1\") "
Jul 16 22:43:45.025286 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.025261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf" (OuterVolumeSpecName: "kube-api-access-xzshf") pod "7f507ea7-92ac-459a-b869-3a0dac1cfba1" (UID: "7f507ea7-92ac-459a-b869-3a0dac1cfba1"). InnerVolumeSpecName "kube-api-access-xzshf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:45.069575 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.069548    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:45.070049 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.070036    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:45.070095 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.070052    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:45.070141 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.070131    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:45.073429 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.073413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.078911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.078888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:45.124777 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.124750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75t2\" (UniqueName: \"kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2\") pod \"test-curl-7472-139852394112320\" (UID: \"491346d0-0f64-4bbf-b606-ff9ab1c8a793\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.124914 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.124797    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzshf\" (UniqueName: \"kubernetes.io/projected/7f507ea7-92ac-459a-b869-3a0dac1cfba1-kube-api-access-xzshf\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:45.225779 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.225746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75t2\" (UniqueName: \"kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2\") pod \"test-curl-7472-139852394112320\" (UID: \"491346d0-0f64-4bbf-b606-ff9ab1c8a793\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.234128 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.234100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75t2\" (UniqueName: \"kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2\") pod \"test-curl-7472-139852394112320\" (UID: \"491346d0-0f64-4bbf-b606-ff9ab1c8a793\") " pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.384046 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.383955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.519516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.519484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:45.521804 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:43:45.521778    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod491346d0_0f64_4bbf_b606_ff9ab1c8a793.slice/crio-5a5863df1f7345e846b1f100aeb359c65a51eb1c28175c70ddd4062b2299ebdf WatchSource:0}: Error finding container 5a5863df1f7345e846b1f100aeb359c65a51eb1c28175c70ddd4062b2299ebdf: Status 404 returned error can't find the container with id 5a5863df1f7345e846b1f100aeb359c65a51eb1c28175c70ddd4062b2299ebdf
Jul 16 22:43:45.765270 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.765243    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:45.765474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.765254    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac5b555474fdff6af589e32088f928ecc550f062f3ba5d4a69665ab27f7299b"
Jul 16 22:43:45.766474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.766437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394112320" event={"ID":"491346d0-0f64-4bbf-b606-ff9ab1c8a793","Type":"ContainerStarted","Data":"5a5863df1f7345e846b1f100aeb359c65a51eb1c28175c70ddd4062b2299ebdf"}
Jul 16 22:43:45.776970 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:45.776939    2566 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7472-139852394112320" oldPodUID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793"
Jul 16 22:43:46.011439 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:46.011407    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f507ea7-92ac-459a-b869-3a0dac1cfba1" path="/var/lib/kubelet/pods/7f507ea7-92ac-459a-b869-3a0dac1cfba1/volumes"
Jul 16 22:43:46.773657 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:46.773626    2566 generic.go:356] "Generic (PLEG): container finished" podID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" containerID="50b22fdaa43ef9ab0012c63497b4e41a07ac5f35b4a2398be061b09a861df875" exitCode=0
Jul 16 22:43:46.773822 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:46.773673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394112320" event={"ID":"491346d0-0f64-4bbf-b606-ff9ab1c8a793","Type":"ContainerDied","Data":"50b22fdaa43ef9ab0012c63497b4e41a07ac5f35b4a2398be061b09a861df875"}
Jul 16 22:43:47.916160 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:47.916134    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:47.951468 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:47.951418    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852394112320_491346d0-0f64-4bbf-b606-ff9ab1c8a793/test-curl-7472-139852394112320/0.log"
Jul 16 22:43:48.001328 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.001277    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:48.014314 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.014280    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852394112320"]
Jul 16 22:43:48.051500 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.051410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d75t2\" (UniqueName: \"kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2\") pod \"491346d0-0f64-4bbf-b606-ff9ab1c8a793\" (UID: \"491346d0-0f64-4bbf-b606-ff9ab1c8a793\") "
Jul 16 22:43:48.053482 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.053434    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2" (OuterVolumeSpecName: "kube-api-access-d75t2") pod "491346d0-0f64-4bbf-b606-ff9ab1c8a793" (UID: "491346d0-0f64-4bbf-b606-ff9ab1c8a793"). InnerVolumeSpecName "kube-api-access-d75t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:48.152914 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.152882    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75t2\" (UniqueName: \"kubernetes.io/projected/491346d0-0f64-4bbf-b606-ff9ab1c8a793-kube-api-access-d75t2\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:48.156809 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.156782    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:48.157225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.157212    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:48.157274 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.157226    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:48.157312 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.157303    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" containerName="test-curl-7472-139852394112320"
Jul 16 22:43:48.160423 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.160407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:48.174211 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.174180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:48.253388 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.253352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4q8\" (UniqueName: \"kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8\") pod \"test-curl-7472-139852394113328\" (UID: \"d4bb3327-7565-4c97-96f3-bf3a46abde02\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:48.354600 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.354509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4q8\" (UniqueName: \"kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8\") pod \"test-curl-7472-139852394113328\" (UID: \"d4bb3327-7565-4c97-96f3-bf3a46abde02\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:48.372153 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.372127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4q8\" (UniqueName: \"kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8\") pod \"test-curl-7472-139852394113328\" (UID: \"d4bb3327-7565-4c97-96f3-bf3a46abde02\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:48.469087 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.469054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:48.614924 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.614842    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:48.617963 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:43:48.617936    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4bb3327_7565_4c97_96f3_bf3a46abde02.slice/crio-3f1d95fa50d2dfd5bdc91956ac0e19271c7fce091f0b07394d1cc9a3b5a0ef30 WatchSource:0}: Error finding container 3f1d95fa50d2dfd5bdc91956ac0e19271c7fce091f0b07394d1cc9a3b5a0ef30: Status 404 returned error can't find the container with id 3f1d95fa50d2dfd5bdc91956ac0e19271c7fce091f0b07394d1cc9a3b5a0ef30
Jul 16 22:43:48.786579 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.786549    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:48.786579 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.786569    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a5863df1f7345e846b1f100aeb359c65a51eb1c28175c70ddd4062b2299ebdf"
Jul 16 22:43:48.787708 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:48.787680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394113328" event={"ID":"d4bb3327-7565-4c97-96f3-bf3a46abde02","Type":"ContainerStarted","Data":"3f1d95fa50d2dfd5bdc91956ac0e19271c7fce091f0b07394d1cc9a3b5a0ef30"}
Jul 16 22:43:48.803304 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:43:48.803280    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7472-139852394112320\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:49.795515 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:49.795479    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4bb3327-7565-4c97-96f3-bf3a46abde02" containerID="bb018d72c2fd6394740e39cfec393eca4a5f576c8e01d06bd016434d5fd6a6f3" exitCode=0
Jul 16 22:43:49.795886 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:49.795567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394113328" event={"ID":"d4bb3327-7565-4c97-96f3-bf3a46abde02","Type":"ContainerDied","Data":"bb018d72c2fd6394740e39cfec393eca4a5f576c8e01d06bd016434d5fd6a6f3"}
Jul 16 22:43:49.817264 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:43:49.817234    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7472-139852394112320\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" pod="opendatahub/test-curl-7472-139852394112320"
Jul 16 22:43:50.011076 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.011042    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491346d0-0f64-4bbf-b606-ff9ab1c8a793" path="/var/lib/kubelet/pods/491346d0-0f64-4bbf-b606-ff9ab1c8a793/volumes"
Jul 16 22:43:50.110640 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.110441    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" podUID="87559132-785e-43c9-9b28-b85b2e26553f" containerName="main" containerID="cri-o://e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f" gracePeriod=45
Jul 16 22:43:50.365147 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.365091    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:50.473319 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmkq5\" (UniqueName: \"kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473503 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache\") pod \"87559132-785e-43c9-9b28-b85b2e26553f\" (UID: \"87559132-785e-43c9-9b28-b85b2e26553f\") "
Jul 16 22:43:50.473670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:50.473670 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home" (OuterVolumeSpecName: "home") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:50.473870 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache" (OuterVolumeSpecName: "model-cache") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:50.473915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473887    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-tmp-dir\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.473915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.473908    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-home\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.475649 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.475619    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm" (OuterVolumeSpecName: "dshm") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:50.475753 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.475677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5" (OuterVolumeSpecName: "kube-api-access-jmkq5") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "kube-api-access-jmkq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:50.475753 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.475694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87559132-785e-43c9-9b28-b85b2e26553f" (UID: "87559132-785e-43c9-9b28-b85b2e26553f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:43:50.574643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.574606    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmkq5\" (UniqueName: \"kubernetes.io/projected/87559132-785e-43c9-9b28-b85b2e26553f-kube-api-access-jmkq5\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.574643 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.574636    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87559132-785e-43c9-9b28-b85b2e26553f-tls-certs\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.574816 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.574651    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-dshm\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.574816 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.574662    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87559132-785e-43c9-9b28-b85b2e26553f-model-cache\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.802342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.802309    2566 generic.go:356] "Generic (PLEG): container finished" podID="87559132-785e-43c9-9b28-b85b2e26553f" containerID="e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f" exitCode=0
Jul 16 22:43:50.802759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.802348    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"
Jul 16 22:43:50.802759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.802379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" event={"ID":"87559132-785e-43c9-9b28-b85b2e26553f","Type":"ContainerDied","Data":"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"}
Jul 16 22:43:50.802759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.802409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5" event={"ID":"87559132-785e-43c9-9b28-b85b2e26553f","Type":"ContainerDied","Data":"bc577f21b4bbf6222ec92f886dc9d15ad9fbc66f681be175b9664cc09aa5dea3"}
Jul 16 22:43:50.802759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.802425    2566 scope.go:117] "RemoveContainer" containerID="e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"
Jul 16 22:43:50.813586 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.813563    2566 scope.go:117] "RemoveContainer" containerID="e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"
Jul 16 22:43:50.813906 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:43:50.813876    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f\": container with ID starting with e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f not found: ID does not exist" containerID="e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"
Jul 16 22:43:50.813998 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.813925    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f"} err="failed to get container status \"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f\": rpc error: code = NotFound desc = could not find container \"e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f\": container with ID starting with e06e93e885fe9b0faa14fd9d28fed19d27ef8b45cb381ae00671a095ef0a363f not found: ID does not exist"
Jul 16 22:43:50.824837 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.824814    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:43:50.830916 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.830892    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7d2f73/rate-test-model-7d2f73-kserve-bfd5d486c-pzct5"]
Jul 16 22:43:50.928926 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.928902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:50.950771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.950705    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852394113328_d4bb3327-7565-4c97-96f3-bf3a46abde02/test-curl-7472-139852394113328/0.log"
Jul 16 22:43:50.976337 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.976311    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:50.980996 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:50.980974    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:51.079582 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.079521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c4q8\" (UniqueName: \"kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8\") pod \"d4bb3327-7565-4c97-96f3-bf3a46abde02\" (UID: \"d4bb3327-7565-4c97-96f3-bf3a46abde02\") "
Jul 16 22:43:51.081489 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.081468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8" (OuterVolumeSpecName: "kube-api-access-5c4q8") pod "d4bb3327-7565-4c97-96f3-bf3a46abde02" (UID: "d4bb3327-7565-4c97-96f3-bf3a46abde02"). InnerVolumeSpecName "kube-api-access-5c4q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:51.107270 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107246    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:51.107705 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107692    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87559132-785e-43c9-9b28-b85b2e26553f" containerName="main"
Jul 16 22:43:51.107755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107707    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87559132-785e-43c9-9b28-b85b2e26553f" containerName="main"
Jul 16 22:43:51.107755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107717    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bb3327-7565-4c97-96f3-bf3a46abde02" containerName="test-curl-7472-139852394113328"
Jul 16 22:43:51.107755 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bb3327-7565-4c97-96f3-bf3a46abde02" containerName="test-curl-7472-139852394113328"
Jul 16 22:43:51.107848 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107794    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bb3327-7565-4c97-96f3-bf3a46abde02" containerName="test-curl-7472-139852394113328"
Jul 16 22:43:51.107848 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.107804    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87559132-785e-43c9-9b28-b85b2e26553f" containerName="main"
Jul 16 22:43:51.110877 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.110863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.116001 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.115981    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:51.180360 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.180332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ddb\" (UniqueName: \"kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb\") pod \"test-curl-7472-139852394113328\" (UID: \"3d7a6b9f-e03e-45d5-8677-03e26b58450f\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.180518 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.180497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4q8\" (UniqueName: \"kubernetes.io/projected/d4bb3327-7565-4c97-96f3-bf3a46abde02-kube-api-access-5c4q8\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:51.281419 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.281385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ddb\" (UniqueName: \"kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb\") pod \"test-curl-7472-139852394113328\" (UID: \"3d7a6b9f-e03e-45d5-8677-03e26b58450f\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.289781 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.289758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ddb\" (UniqueName: \"kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb\") pod \"test-curl-7472-139852394113328\" (UID: \"3d7a6b9f-e03e-45d5-8677-03e26b58450f\") " pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.434623 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.434544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.772397 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.772375    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:51.775064 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:43:51.775034    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d7a6b9f_e03e_45d5_8677_03e26b58450f.slice/crio-11bd8609f28ddde8181cb5bdf1caf5b4a247ab9107f12fbec2b814ecfc60ff5e WatchSource:0}: Error finding container 11bd8609f28ddde8181cb5bdf1caf5b4a247ab9107f12fbec2b814ecfc60ff5e: Status 404 returned error can't find the container with id 11bd8609f28ddde8181cb5bdf1caf5b4a247ab9107f12fbec2b814ecfc60ff5e
Jul 16 22:43:51.810375 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.810347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394113328" event={"ID":"3d7a6b9f-e03e-45d5-8677-03e26b58450f","Type":"ContainerStarted","Data":"11bd8609f28ddde8181cb5bdf1caf5b4a247ab9107f12fbec2b814ecfc60ff5e"}
Jul 16 22:43:51.811771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.811688    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f1d95fa50d2dfd5bdc91956ac0e19271c7fce091f0b07394d1cc9a3b5a0ef30"
Jul 16 22:43:51.811771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.811717    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:51.823197 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:51.823174    2566 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7472-139852394113328" oldPodUID="d4bb3327-7565-4c97-96f3-bf3a46abde02" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f"
Jul 16 22:43:52.011367 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:52.011333    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87559132-785e-43c9-9b28-b85b2e26553f" path="/var/lib/kubelet/pods/87559132-785e-43c9-9b28-b85b2e26553f/volumes"
Jul 16 22:43:52.011678 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:52.011663    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4bb3327-7565-4c97-96f3-bf3a46abde02" path="/var/lib/kubelet/pods/d4bb3327-7565-4c97-96f3-bf3a46abde02/volumes"
Jul 16 22:43:52.818698 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:52.818659    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" containerID="328c43a11b0227aa1ce96731efbd284a2c3dc10f4fb20d7a529d2ad58a143c3c" exitCode=0
Jul 16 22:43:52.819074 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:52.818707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7472-139852394113328" event={"ID":"3d7a6b9f-e03e-45d5-8677-03e26b58450f","Type":"ContainerDied","Data":"328c43a11b0227aa1ce96731efbd284a2c3dc10f4fb20d7a529d2ad58a143c3c"}
Jul 16 22:43:53.963027 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:53.963002    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:53.986641 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:53.986617    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7472-139852394113328_3d7a6b9f-e03e-45d5-8677-03e26b58450f/test-curl-7472-139852394113328/0.log"
Jul 16 22:43:54.014705 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.014678    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:54.018273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.018247    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7472-139852394113328"]
Jul 16 22:43:54.109759 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.109678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9ddb\" (UniqueName: \"kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb\") pod \"3d7a6b9f-e03e-45d5-8677-03e26b58450f\" (UID: \"3d7a6b9f-e03e-45d5-8677-03e26b58450f\") "
Jul 16 22:43:54.111709 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.111678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb" (OuterVolumeSpecName: "kube-api-access-k9ddb") pod "3d7a6b9f-e03e-45d5-8677-03e26b58450f" (UID: "3d7a6b9f-e03e-45d5-8677-03e26b58450f"). InnerVolumeSpecName "kube-api-access-k9ddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:54.211034 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.211001    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ddb\" (UniqueName: \"kubernetes.io/projected/3d7a6b9f-e03e-45d5-8677-03e26b58450f-kube-api-access-k9ddb\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:43:54.831295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.831267    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bd8609f28ddde8181cb5bdf1caf5b4a247ab9107f12fbec2b814ecfc60ff5e"
Jul 16 22:43:54.831295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:54.831281    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:54.841566 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:43:54.841537    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7472-139852394113328\" is forbidden: User \"system:node:ip-10-0-140-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-81.ec2.internal' and this object" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" pod="opendatahub/test-curl-7472-139852394113328"
Jul 16 22:43:56.016474 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:43:56.016413    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" path="/var/lib/kubelet/pods/3d7a6b9f-e03e-45d5-8677-03e26b58450f/volumes"
Jul 16 22:44:04.991700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:04.991664    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:44:04.992086 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:04.991882    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" podUID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" containerName="maas-api" containerID="cri-o://2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e" gracePeriod=30
Jul 16 22:44:05.256008 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.255949    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:44:05.417935 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.417906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls\") pod \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") "
Jul 16 22:44:05.418106 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.417977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca\") pod \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") "
Jul 16 22:44:05.418106 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.418029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnk7d\" (UniqueName: \"kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d\") pod \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\" (UID: \"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd\") "
Jul 16 22:44:05.418371 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.418346    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" (UID: "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:05.420046 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.420023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" (UID: "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:05.420143 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.420046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d" (OuterVolumeSpecName: "kube-api-access-nnk7d") pod "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" (UID: "20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd"). InnerVolumeSpecName "kube-api-access-nnk7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:05.519620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.519563    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:05.519620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.519585    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnk7d\" (UniqueName: \"kubernetes.io/projected/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-kube-api-access-nnk7d\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:05.519620 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.519595    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:05.891548 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.891436    2566 generic.go:356] "Generic (PLEG): container finished" podID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" containerID="2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e" exitCode=0
Jul 16 22:44:05.891548 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.891499    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"
Jul 16 22:44:05.891548 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.891525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" event={"ID":"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd","Type":"ContainerDied","Data":"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"}
Jul 16 22:44:05.891815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.891593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825" event={"ID":"20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd","Type":"ContainerDied","Data":"c36087de963b6805d7269fb65de9ef856feaf988a8e70878eb6ad1bfd612ef60"}
Jul 16 22:44:05.891815 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.891627    2566 scope.go:117] "RemoveContainer" containerID="2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"
Jul 16 22:44:05.901365 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.901347    2566 scope.go:117] "RemoveContainer" containerID="2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"
Jul 16 22:44:05.901632 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:05.901610    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e\": container with ID starting with 2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e not found: ID does not exist" containerID="2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"
Jul 16 22:44:05.901758 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.901644    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e"} err="failed to get container status \"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e\": rpc error: code = NotFound desc = could not find container \"2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e\": container with ID starting with 2d295783dd5fcef332a64c98f4b92396077eb3862519f9e3b9944d16ece0bd2e not found: ID does not exist"
Jul 16 22:44:05.918138 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.918115    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:44:05.921598 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:05.921574    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c9596-5b456c4c55-ww825"]
Jul 16 22:44:06.011764 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:06.011740    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" path="/var/lib/kubelet/pods/20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd/volumes"
Jul 16 22:44:31.146935 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.146893    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:44:31.147669 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.147153    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" podUID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" containerName="maas-api" containerID="cri-o://776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a" gracePeriod=30
Jul 16 22:44:31.402939 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.402884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:44:31.456991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.456959    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls\") pod \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") "
Jul 16 22:44:31.457148 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.457051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlbvg\" (UniqueName: \"kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg\") pod \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") "
Jul 16 22:44:31.457148 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.457081    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca\") pod \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\" (UID: \"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a\") "
Jul 16 22:44:31.457499 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.457470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" (UID: "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:31.458999 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.458977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" (UID: "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:31.459131 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.459114    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg" (OuterVolumeSpecName: "kube-api-access-jlbvg") pod "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" (UID: "0bff337d-99eb-45f0-94a3-9c9dc5b87e0a"). InnerVolumeSpecName "kube-api-access-jlbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:31.558098 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.558064    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-maas-api-tls\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:31.558098 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.558093    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlbvg\" (UniqueName: \"kubernetes.io/projected/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-kube-api-access-jlbvg\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:31.558098 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:31.558104    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a-openshift-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:32.033185 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.033150    2566 generic.go:356] "Generic (PLEG): container finished" podID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" containerID="776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a" exitCode=0
Jul 16 22:44:32.033346 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.033202    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"
Jul 16 22:44:32.033346 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.033234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" event={"ID":"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a","Type":"ContainerDied","Data":"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"}
Jul 16 22:44:32.033346 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.033288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj" event={"ID":"0bff337d-99eb-45f0-94a3-9c9dc5b87e0a","Type":"ContainerDied","Data":"bd4576579f3884d1cbe069ee3da447b3351b187ce558df9fa0ef7c44832fca74"}
Jul 16 22:44:32.033346 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.033314    2566 scope.go:117] "RemoveContainer" containerID="776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"
Jul 16 22:44:32.042193 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.042176    2566 scope.go:117] "RemoveContainer" containerID="776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"
Jul 16 22:44:32.042488 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:32.042466    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a\": container with ID starting with 776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a not found: ID does not exist" containerID="776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"
Jul 16 22:44:32.042551 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.042497    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a"} err="failed to get container status \"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a\": rpc error: code = NotFound desc = could not find container \"776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a\": container with ID starting with 776b33d3cece42a32a689b6702281b7fec2caee67bb9d9d1c091ba7ec410db9a not found: ID does not exist"
Jul 16 22:44:32.051478 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.051444    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:44:32.057896 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:32.057875    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7d2f73-99fbb558d-67zvj"]
Jul 16 22:44:34.011622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:34.011579    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" path="/var/lib/kubelet/pods/0bff337d-99eb-45f0-94a3-9c9dc5b87e0a/volumes"
Jul 16 22:44:36.070292 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070252    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:36.070892 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070871    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" containerName="maas-api"
Jul 16 22:44:36.070986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070896    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" containerName="maas-api"
Jul 16 22:44:36.070986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070928    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" containerName="maas-api"
Jul 16 22:44:36.070986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070937    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" containerName="maas-api"
Jul 16 22:44:36.070986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070958    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" containerName="test-curl-7472-139852394113328"
Jul 16 22:44:36.070986 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.070967    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" containerName="test-curl-7472-139852394113328"
Jul 16 22:44:36.071219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.071075    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4deaa-1365-4bbd-89fc-dd05b3bfa6bd" containerName="maas-api"
Jul 16 22:44:36.071219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.071089    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7a6b9f-e03e-45d5-8677-03e26b58450f" containerName="test-curl-7472-139852394113328"
Jul 16 22:44:36.071219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.071103    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bff337d-99eb-45f0-94a3-9c9dc5b87e0a" containerName="maas-api"
Jul 16 22:44:36.074463 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.074431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:36.077306 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.077285    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:44:36.083434 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.083412    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:36.203009 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.202972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8\" (UID: \"22fb25a2-26b5-43a8-92bb-4cc0016abe83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:36.304191 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.304154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8\" (UID: \"22fb25a2-26b5-43a8-92bb-4cc0016abe83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:36.304774 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.304756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8\" (UID: \"22fb25a2-26b5-43a8-92bb-4cc0016abe83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:36.385960 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.385874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:36.522936 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.522912    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:36.525012 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:44:36.524985    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fb25a2_26b5_43a8_92bb_4cc0016abe83.slice/crio-0efdf4c9031805a88e24eed4c8a48ec96520334ee0b41bc2c875f5aa07438633 WatchSource:0}: Error finding container 0efdf4c9031805a88e24eed4c8a48ec96520334ee0b41bc2c875f5aa07438633: Status 404 returned error can't find the container with id 0efdf4c9031805a88e24eed4c8a48ec96520334ee0b41bc2c875f5aa07438633
Jul 16 22:44:36.781837 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.781798    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:36.786581 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.786565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:36.799887 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.799849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:36.910180 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:36.910151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr\" (UID: \"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:37.010749 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.010674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr\" (UID: \"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:37.011228 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.011209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr\" (UID: \"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:37.062745 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.062711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerStarted","Data":"e094d7b60dea67a364cef6e62ae9f1e1b9075ea71c9d96fa1f5ce4c20ed484e6"}
Jul 16 22:44:37.062901 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.062771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerStarted","Data":"0efdf4c9031805a88e24eed4c8a48ec96520334ee0b41bc2c875f5aa07438633"}
Jul 16 22:44:37.080723 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.080677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" podStartSLOduration=1.080661766 podStartE2EDuration="1.080661766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:37.0789391 +0000 UTC m=+3435.588806181" watchObservedRunningTime="2026-07-16 22:44:37.080661766 +0000 UTC m=+3435.590528847"
Jul 16 22:44:37.096384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.096354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:37.433638 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:37.433606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:37.435035 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:44:37.435011    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode716aa6b_78bf_41e4_b1e1_b0bcb93e32ef.slice/crio-fa490ebc52eb86ae756140fd62850d561e8c57235b5f615ed972acfbf9a50a34 WatchSource:0}: Error finding container fa490ebc52eb86ae756140fd62850d561e8c57235b5f615ed972acfbf9a50a34: Status 404 returned error can't find the container with id fa490ebc52eb86ae756140fd62850d561e8c57235b5f615ed972acfbf9a50a34
Jul 16 22:44:38.070046 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.070021    2566 generic.go:356] "Generic (PLEG): container finished" podID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerID="293e72a68a170fe25fbe1539b01682635f83d47edeaeb70cb1316b5d999fd2f6" exitCode=6
Jul 16 22:44:38.070175 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.070111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" event={"ID":"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef","Type":"ContainerDied","Data":"293e72a68a170fe25fbe1539b01682635f83d47edeaeb70cb1316b5d999fd2f6"}
Jul 16 22:44:38.070238 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.070172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" event={"ID":"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef","Type":"ContainerStarted","Data":"fa490ebc52eb86ae756140fd62850d561e8c57235b5f615ed972acfbf9a50a34"}
Jul 16 22:44:38.070289 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.070265    2566 scope.go:117] "RemoveContainer" containerID="293e72a68a170fe25fbe1539b01682635f83d47edeaeb70cb1316b5d999fd2f6"
Jul 16 22:44:38.071741 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.071720    2566 generic.go:356] "Generic (PLEG): container finished" podID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerID="e094d7b60dea67a364cef6e62ae9f1e1b9075ea71c9d96fa1f5ce4c20ed484e6" exitCode=6
Jul 16 22:44:38.071838 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.071762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerDied","Data":"e094d7b60dea67a364cef6e62ae9f1e1b9075ea71c9d96fa1f5ce4c20ed484e6"}
Jul 16 22:44:38.071972 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:38.071960    2566 scope.go:117] "RemoveContainer" containerID="e094d7b60dea67a364cef6e62ae9f1e1b9075ea71c9d96fa1f5ce4c20ed484e6"
Jul 16 22:44:39.078803 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.078765    2566 generic.go:356] "Generic (PLEG): container finished" podID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerID="d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109" exitCode=6
Jul 16 22:44:39.079225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.078849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" event={"ID":"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef","Type":"ContainerDied","Data":"d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109"}
Jul 16 22:44:39.079225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.078931    2566 scope.go:117] "RemoveContainer" containerID="293e72a68a170fe25fbe1539b01682635f83d47edeaeb70cb1316b5d999fd2f6"
Jul 16 22:44:39.079225 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.079021    2566 scope.go:117] "RemoveContainer" containerID="d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109"
Jul 16 22:44:39.079367 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:39.079272    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr_odh-ai-gateway-infra(e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef"
Jul 16 22:44:39.080877 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.080856    2566 generic.go:356] "Generic (PLEG): container finished" podID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerID="ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c" exitCode=6
Jul 16 22:44:39.080967 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.080894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerDied","Data":"ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c"}
Jul 16 22:44:39.081100 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.081088    2566 scope.go:117] "RemoveContainer" containerID="ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c"
Jul 16 22:44:39.081264 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:39.081248    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8_odh-ai-gateway-infra(22fb25a2-26b5-43a8-92bb-4cc0016abe83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83"
Jul 16 22:44:39.090229 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:39.090213    2566 scope.go:117] "RemoveContainer" containerID="e094d7b60dea67a364cef6e62ae9f1e1b9075ea71c9d96fa1f5ce4c20ed484e6"
Jul 16 22:44:40.088848 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:40.088820    2566 scope.go:117] "RemoveContainer" containerID="d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109"
Jul 16 22:44:40.089245 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:40.089029    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr_odh-ai-gateway-infra(e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef"
Jul 16 22:44:40.090251 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:40.090229    2566 scope.go:117] "RemoveContainer" containerID="ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c"
Jul 16 22:44:40.090468 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:40.090437    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8_odh-ai-gateway-infra(22fb25a2-26b5-43a8-92bb-4cc0016abe83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83"
Jul 16 22:44:53.005304 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.005270    2566 scope.go:117] "RemoveContainer" containerID="d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109"
Jul 16 22:44:53.007806 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.005734    2566 scope.go:117] "RemoveContainer" containerID="ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c"
Jul 16 22:44:53.282773 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.282714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w5c8b_f938169b-f9fb-4385-a27c-4e7450ed1366/manager/0.log"
Jul 16 22:44:53.404464 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.404428    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-74qf2_bb8353bb-5367-4dc1-8d85-3cead16b9d5b/manager/0.log"
Jul 16 22:44:53.528891 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.528865    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58f7bcf9f5-sxhf9_b555e970-cac6-4667-8742-523d8be45e71/manager/0.log"
Jul 16 22:44:53.770054 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.770028    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nlx76_75ce86e8-4a0f-4bed-99cc-8a206c2216cf/manager/0.log"
Jul 16 22:44:53.890273 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.890201    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-5jrpx_40c0974e-151e-40e5-a6c3-bad33f4c32b0/copy-manifests/0.log"
Jul 16 22:44:53.903496 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:53.903448    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-5jrpx_40c0974e-151e-40e5-a6c3-bad33f4c32b0/manager/0.log"
Jul 16 22:44:54.167427 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.167351    2566 generic.go:356] "Generic (PLEG): container finished" podID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerID="a7951dea4af392ab22e9a25e1106c4044c8e2dde4b4e790b40f786fba90578b2" exitCode=6
Jul 16 22:44:54.167851 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.167424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" event={"ID":"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef","Type":"ContainerDied","Data":"a7951dea4af392ab22e9a25e1106c4044c8e2dde4b4e790b40f786fba90578b2"}
Jul 16 22:44:54.167851 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.167500    2566 scope.go:117] "RemoveContainer" containerID="d0d2bb83617c14cc1453b0ba04e6bff5aaf30b5b33e96d7fb45b928cb866e109"
Jul 16 22:44:54.167851 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.167817    2566 scope.go:117] "RemoveContainer" containerID="a7951dea4af392ab22e9a25e1106c4044c8e2dde4b4e790b40f786fba90578b2"
Jul 16 22:44:54.168063 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:54.168041    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr_odh-ai-gateway-infra(e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef"
Jul 16 22:44:54.169393 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.169373    2566 generic.go:356] "Generic (PLEG): container finished" podID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerID="bfe7b0b84c274361990181a09672266ea2726531ef9571a222590f734e425887" exitCode=6
Jul 16 22:44:54.169512 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.169422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerDied","Data":"bfe7b0b84c274361990181a09672266ea2726531ef9571a222590f734e425887"}
Jul 16 22:44:54.169693 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.169680    2566 scope.go:117] "RemoveContainer" containerID="bfe7b0b84c274361990181a09672266ea2726531ef9571a222590f734e425887"
Jul 16 22:44:54.169849 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:44:54.169833    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8_odh-ai-gateway-infra(22fb25a2-26b5-43a8-92bb-4cc0016abe83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83"
Jul 16 22:44:54.179083 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.179068    2566 scope.go:117] "RemoveContainer" containerID="ed409b4ff9212804267c58396695ae5fa10b174e3bd6a5f5f14c8658d0fdda7c"
Jul 16 22:44:54.538432 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.538404    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77c4c984f4-pqbgm_5510fa8c-ea98-4a48-87fb-2f00c661465f/maas-api/0.log"
Jul 16 22:44:54.656420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.656389    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737320-4q46q_e666691e-b48f-4e70-8123-7418cd485a74/cleanup/0.log"
Jul 16 22:44:54.769619 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.769584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737335-fdqjr_5557b882-29fd-44a3-bd5a-9c97ebccf7f6/cleanup/0.log"
Jul 16 22:44:54.874472 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.874380    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737350-k2vfg_ff39cd25-720b-41d0-a8cf-035226b297a5/cleanup/0.log"
Jul 16 22:44:54.984018 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:54.983983    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr_e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef/revoke-keys/2.log"
Jul 16 22:44:55.194356 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.194256    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:55.210663 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.210635    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:55.345865 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.345832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:55.374243 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.374218    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:55.470288 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.470224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca\") pod \"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef\" (UID: \"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef\") "
Jul 16 22:44:55.470288 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.470266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca\") pod \"22fb25a2-26b5-43a8-92bb-4cc0016abe83\" (UID: \"22fb25a2-26b5-43a8-92bb-4cc0016abe83\") "
Jul 16 22:44:55.470596 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.470575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" (UID: "e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:55.470658 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.470639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22fb25a2-26b5-43a8-92bb-4cc0016abe83" (UID: "22fb25a2-26b5-43a8-92bb-4cc0016abe83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:55.557587 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.557564    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8_22fb25a2-26b5-43a8-92bb-4cc0016abe83/revoke-keys/2.log"
Jul 16 22:44:55.570973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.570951    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:55.570973 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:55.570971    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22fb25a2-26b5-43a8-92bb-4cc0016abe83-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:44:56.106859 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.106835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-cpgdw_fc94e78e-1fee-4eb4-a2d1-a0709a7d5c25/postgres/0.log"
Jul 16 22:44:56.183854 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.183828    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"
Jul 16 22:44:56.184013 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.183864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr" event={"ID":"e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef","Type":"ContainerDied","Data":"fa490ebc52eb86ae756140fd62850d561e8c57235b5f615ed972acfbf9a50a34"}
Jul 16 22:44:56.184013 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.183953    2566 scope.go:117] "RemoveContainer" containerID="a7951dea4af392ab22e9a25e1106c4044c8e2dde4b4e790b40f786fba90578b2"
Jul 16 22:44:56.185214 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.185190    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"
Jul 16 22:44:56.185317 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.185194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8" event={"ID":"22fb25a2-26b5-43a8-92bb-4cc0016abe83","Type":"ContainerDied","Data":"0efdf4c9031805a88e24eed4c8a48ec96520334ee0b41bc2c875f5aa07438633"}
Jul 16 22:44:56.192827 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.192811    2566 scope.go:117] "RemoveContainer" containerID="bfe7b0b84c274361990181a09672266ea2726531ef9571a222590f734e425887"
Jul 16 22:44:56.201513 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.201474    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:56.209127 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.209106    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-5xgjr"]
Jul 16 22:44:56.222647 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.222627    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:56.227395 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:56.227371    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f123e265-ldgk8"]
Jul 16 22:44:57.486090 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.486062    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-648pc_0407b5d5-b43d-4dfc-8901-c1b0a68a9c68/manager/0.log"
Jul 16 22:44:57.600567 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.600536    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mjwvd_bb659723-9fd2-4429-995d-693d040a8151/manager/0.log"
Jul 16 22:44:57.706297 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.706274    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qlxl2_1ddb94a5-fff7-46e9-bdcc-22818765e0b2/kuadrant-console-plugin/0.log"
Jul 16 22:44:57.821915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.821887    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-nv9jj_edfa8562-85ec-4ed4-911d-29ebc0e243da/registry-server/0.log"
Jul 16 22:44:57.873665 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.873633    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:44:57.874072 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874059    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874073    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874094    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874101    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874116    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874119 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874121    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874325 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874191    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874325 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874200    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:44:57.874325 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874205    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.874325 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.874211    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:44:57.878684 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.878668    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:44:57.881215 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.881196    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:44:57.885409 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.885315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:44:57.995735 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:57.995697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c\" (UID: \"a95f3b78-597b-43f7-b4d2-6276f19afefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:44:58.011056 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.011023    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" path="/var/lib/kubelet/pods/22fb25a2-26b5-43a8-92bb-4cc0016abe83/volumes"
Jul 16 22:44:58.011385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.011371    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" path="/var/lib/kubelet/pods/e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef/volumes"
Jul 16 22:44:58.096884 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.096797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c\" (UID: \"a95f3b78-597b-43f7-b4d2-6276f19afefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:44:58.097499 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.097441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c\" (UID: \"a95f3b78-597b-43f7-b4d2-6276f19afefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:44:58.190374 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.190343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:44:58.200845 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.200822    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kt4t9_1639d5a9-19bf-433d-a250-02c54261fc78/manager/0.log"
Jul 16 22:44:58.343563 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.343540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:44:58.346015 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:44:58.345990    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95f3b78_597b_43f7_b4d2_6276f19afefa.slice/crio-7187c0cb84fa9dc312ecf4dc2c15ee4bbeca32226b3e685722e60c8b479b61ef WatchSource:0}: Error finding container 7187c0cb84fa9dc312ecf4dc2c15ee4bbeca32226b3e685722e60c8b479b61ef: Status 404 returned error can't find the container with id 7187c0cb84fa9dc312ecf4dc2c15ee4bbeca32226b3e685722e60c8b479b61ef
Jul 16 22:44:58.576382 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.576356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47ptlz_dc34873d-40f2-4153-a382-d69f6c703405/istio-proxy/0.log"
Jul 16 22:44:58.818814 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:58.818784    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-2tn7p_d303fb66-62db-49a0-afce-89f3313d9ec6/kube-auth-proxy/0.log"
Jul 16 22:44:59.066268 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.066238    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/1.log"
Jul 16 22:44:59.202406 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.202305    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-6jvh5_b65f15b0-318e-41ba-8fbd-6bb7d008343d/payload-pre-processing/0.log"
Jul 16 22:44:59.206744 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.206713    2566 generic.go:356] "Generic (PLEG): container finished" podID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerID="8c32d1ec99556cf2cd5446d60f058226bce12d307691ff9a58a0cc61ee1d03a0" exitCode=6
Jul 16 22:44:59.206889 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.206797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" event={"ID":"a95f3b78-597b-43f7-b4d2-6276f19afefa","Type":"ContainerDied","Data":"8c32d1ec99556cf2cd5446d60f058226bce12d307691ff9a58a0cc61ee1d03a0"}
Jul 16 22:44:59.206889 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.206849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" event={"ID":"a95f3b78-597b-43f7-b4d2-6276f19afefa","Type":"ContainerStarted","Data":"7187c0cb84fa9dc312ecf4dc2c15ee4bbeca32226b3e685722e60c8b479b61ef"}
Jul 16 22:44:59.206990 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.206974    2566 scope.go:117] "RemoveContainer" containerID="8c32d1ec99556cf2cd5446d60f058226bce12d307691ff9a58a0cc61ee1d03a0"
Jul 16 22:44:59.327363 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.327326    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6s658_5c2dae41-16b6-463c-a7b1-8ce8579a0c60/payload-processing/0.log"
Jul 16 22:44:59.487707 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.487634    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddb94bc54-9kx9k_e4b0774f-37d5-498e-a1ff-69f5189b5398/router/0.log"
Jul 16 22:44:59.857964 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:44:59.857935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bqxxh_bacae03c-47ad-4ace-889f-5db2c21a623f/main/0.log"
Jul 16 22:45:00.111354 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.111269    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9tddb_948df7eb-6796-4896-877e-c3f7c46ae6c1/main/0.log"
Jul 16 22:45:00.189516 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.189476    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"]
Jul 16 22:45:00.190127 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190105    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:45:00.190127 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190125    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:45:00.190262 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190139    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:45:00.190262 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190147    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:45:00.190262 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190223    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e716aa6b-78bf-41e4-b1e1-b0bcb93e32ef" containerName="revoke-keys"
Jul 16 22:45:00.190262 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.190237    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fb25a2-26b5-43a8-92bb-4cc0016abe83" containerName="revoke-keys"
Jul 16 22:45:00.193404 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.193388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"
Jul 16 22:45:00.202421 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.202404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"
Jul 16 22:45:00.212398 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.212364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"]
Jul 16 22:45:00.221823 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.221795    2566 generic.go:356] "Generic (PLEG): container finished" podID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerID="8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14" exitCode=6
Jul 16 22:45:00.222033 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.221877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" event={"ID":"a95f3b78-597b-43f7-b4d2-6276f19afefa","Type":"ContainerDied","Data":"8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14"}
Jul 16 22:45:00.222114 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.222097    2566 scope.go:117] "RemoveContainer" containerID="8c32d1ec99556cf2cd5446d60f058226bce12d307691ff9a58a0cc61ee1d03a0"
Jul 16 22:45:00.222172 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.222126    2566 scope.go:117] "RemoveContainer" containerID="8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14"
Jul 16 22:45:00.222415 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:00.222384    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c_odh-ai-gateway-infra(a95f3b78-597b-43f7-b4d2-6276f19afefa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa"
Jul 16 22:45:00.398488 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:00.398395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"]
Jul 16 22:45:00.401678 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:45:00.401649    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ad9769_613d_419a_aa8a_d46f26d7b62c.slice/crio-cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3 WatchSource:0}: Error finding container cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3: Status 404 returned error can't find the container with id cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3
Jul 16 22:45:01.231281 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:01.231232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd" event={"ID":"61ad9769-613d-419a-aa8a-d46f26d7b62c","Type":"ContainerStarted","Data":"c49187dd3bdcf97ede6ed5b5a1115a9ae8b52eaf5c3aa5786a7fe1b9a01fcf7f"}
Jul 16 22:45:01.231736 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:01.231306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd" event={"ID":"61ad9769-613d-419a-aa8a-d46f26d7b62c","Type":"ContainerStarted","Data":"cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3"}
Jul 16 22:45:01.232900 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:01.232878    2566 scope.go:117] "RemoveContainer" containerID="8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14"
Jul 16 22:45:01.233069 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:01.233051    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c_odh-ai-gateway-infra(a95f3b78-597b-43f7-b4d2-6276f19afefa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa"
Jul 16 22:45:01.257766 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:01.257724    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd" podStartSLOduration=1.257712877 podStartE2EDuration="1.257712877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:01.255549259 +0000 UTC m=+3459.765416339" watchObservedRunningTime="2026-07-16 22:45:01.257712877 +0000 UTC m=+3459.767579958"
Jul 16 22:45:02.246166 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:02.246126    2566 generic.go:356] "Generic (PLEG): container finished" podID="61ad9769-613d-419a-aa8a-d46f26d7b62c" containerID="c49187dd3bdcf97ede6ed5b5a1115a9ae8b52eaf5c3aa5786a7fe1b9a01fcf7f" exitCode=0
Jul 16 22:45:02.246568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:02.246181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd" event={"ID":"61ad9769-613d-419a-aa8a-d46f26d7b62c","Type":"ContainerDied","Data":"c49187dd3bdcf97ede6ed5b5a1115a9ae8b52eaf5c3aa5786a7fe1b9a01fcf7f"}
Jul 16 22:45:03.392710 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:03.392685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"
Jul 16 22:45:04.259209 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:04.259181    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd"
Jul 16 22:45:04.259384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:04.259224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-4wxmd" event={"ID":"61ad9769-613d-419a-aa8a-d46f26d7b62c","Type":"ContainerDied","Data":"cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3"}
Jul 16 22:45:04.259384 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:04.259271    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cacefb128364b1c41a01e2ed4a4876f51ebf7f060084d737bb32f1b896171ad3"
Jul 16 22:45:04.337265 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:04.337187    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"]
Jul 16 22:45:04.340515 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:04.340490    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737320-4q46q"]
Jul 16 22:45:06.016007 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:06.015969    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e666691e-b48f-4e70-8123-7418cd485a74" path="/var/lib/kubelet/pods/e666691e-b48f-4e70-8123-7418cd485a74/volumes"
Jul 16 22:45:08.650837 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:08.650807    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mj2sj_9aa83795-0672-4fc2-9607-bbcc4434ece5/global-pull-secret-syncer/0.log"
Jul 16 22:45:08.843029 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:08.842999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tcj9h_78785ffe-be06-4f51-a2e1-a92e54ebcfa4/konnectivity-agent/0.log"
Jul 16 22:45:08.903997 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:08.903901    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-81.ec2.internal_05d12d6b74a3f6fbeca9855716ed7cfc/haproxy/0.log"
Jul 16 22:45:14.446998 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:14.446960    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-648pc_0407b5d5-b43d-4dfc-8901-c1b0a68a9c68/manager/0.log"
Jul 16 22:45:14.475859 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:14.475832    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-mjwvd_bb659723-9fd2-4429-995d-693d040a8151/manager/0.log"
Jul 16 22:45:14.511018 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:14.510999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qlxl2_1ddb94a5-fff7-46e9-bdcc-22818765e0b2/kuadrant-console-plugin/0.log"
Jul 16 22:45:14.580877 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:14.580853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-nv9jj_edfa8562-85ec-4ed4-911d-29ebc0e243da/registry-server/0.log"
Jul 16 22:45:14.861585 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:14.861544    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kt4t9_1639d5a9-19bf-433d-a250-02c54261fc78/manager/0.log"
Jul 16 22:45:15.004475 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:15.004433    2566 scope.go:117] "RemoveContainer" containerID="8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14"
Jul 16 22:45:16.329321 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:16.329286    2566 generic.go:356] "Generic (PLEG): container finished" podID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerID="4555a1f903e3589d74f45fc18fa7fe040a5ffb35e3f3c9ea230204905af3b8de" exitCode=6
Jul 16 22:45:16.329771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:16.329360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" event={"ID":"a95f3b78-597b-43f7-b4d2-6276f19afefa","Type":"ContainerDied","Data":"4555a1f903e3589d74f45fc18fa7fe040a5ffb35e3f3c9ea230204905af3b8de"}
Jul 16 22:45:16.329771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:16.329431    2566 scope.go:117] "RemoveContainer" containerID="8fc88803a6ae3922f90ee01462ed3263a7ee5a0759030b25149649c7ba22fb14"
Jul 16 22:45:16.329771 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:16.329755    2566 scope.go:117] "RemoveContainer" containerID="4555a1f903e3589d74f45fc18fa7fe040a5ffb35e3f3c9ea230204905af3b8de"
Jul 16 22:45:16.329996 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:16.329976    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c_odh-ai-gateway-infra(a95f3b78-597b-43f7-b4d2-6276f19afefa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa"
Jul 16 22:45:16.858963 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:16.858930    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-lx69d_9b2c8f8c-0c6f-4891-b65b-b7d004450d38/cluster-monitoring-operator/0.log"
Jul 16 22:45:17.143324 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.143245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-n9ngj_c9a8c775-a2f3-4838-a4fb-595f042e4493/monitoring-plugin/0.log"
Jul 16 22:45:17.385722 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.385685    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:45:17.396215 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.396141    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swxjg_f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc/node-exporter/0.log"
Jul 16 22:45:17.443507 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.442735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swxjg_f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc/kube-rbac-proxy/0.log"
Jul 16 22:45:17.473709 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.473684    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swxjg_f96b41f0-3a0e-41fc-b7fe-ce62a32fa1cc/init-textfile/0.log"
Jul 16 22:45:17.513276 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.513249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cp4ws_a9a602cf-12f5-42e2-a053-dda828f0b9e7/kube-rbac-proxy-main/0.log"
Jul 16 22:45:17.527743 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.527721    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:45:17.549385 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.549341    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cp4ws_a9a602cf-12f5-42e2-a053-dda828f0b9e7/kube-rbac-proxy-self/0.log"
Jul 16 22:45:17.566706 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.566685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca\") pod \"a95f3b78-597b-43f7-b4d2-6276f19afefa\" (UID: \"a95f3b78-597b-43f7-b4d2-6276f19afefa\") "
Jul 16 22:45:17.567025 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.567003    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a95f3b78-597b-43f7-b4d2-6276f19afefa" (UID: "a95f3b78-597b-43f7-b4d2-6276f19afefa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:17.579735 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.579709    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cp4ws_a9a602cf-12f5-42e2-a053-dda828f0b9e7/openshift-state-metrics/0.log"
Jul 16 22:45:17.667919 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.667861    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a95f3b78-597b-43f7-b4d2-6276f19afefa-maas-api-service-ca\") on node \"ip-10-0-140-81.ec2.internal\" DevicePath \"\""
Jul 16 22:45:17.810588 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.810557    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-574rp_6768fca8-cba6-46e4-9b94-25f48aa790a1/prometheus-operator/0.log"
Jul 16 22:45:17.831929 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.831908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-574rp_6768fca8-cba6-46e4-9b94-25f48aa790a1/kube-rbac-proxy/0.log"
Jul 16 22:45:17.982326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:17.982245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/thanos-query/0.log"
Jul 16 22:45:18.007217 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.007181    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/kube-rbac-proxy-web/0.log"
Jul 16 22:45:18.032656 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.032633    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/kube-rbac-proxy/0.log"
Jul 16 22:45:18.073350 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.073329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/prom-label-proxy/0.log"
Jul 16 22:45:18.107146 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.107125    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/kube-rbac-proxy-rules/0.log"
Jul 16 22:45:18.138073 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.138052    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-798fb6d8b5-xmcng_348f7975-0a2f-4c9f-98d5-e901e6cc75d8/kube-rbac-proxy-metrics/0.log"
Jul 16 22:45:18.307290 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307265    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"]
Jul 16 22:45:18.307700 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307687    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307701    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307709    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ad9769-613d-419a-aa8a-d46f26d7b62c" containerName="cleanup"
Jul 16 22:45:18.307748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307714    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ad9769-613d-419a-aa8a-d46f26d7b62c" containerName="cleanup"
Jul 16 22:45:18.307748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307731    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307748 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307737    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307801    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307810    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:18.307911 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.307817    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ad9769-613d-419a-aa8a-d46f26d7b62c" containerName="cleanup"
Jul 16 22:45:18.311001 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.310983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.315680 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.315659    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"kube-root-ca.crt\""
Jul 16 22:45:18.316123 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.316107    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xsq47\"/\"openshift-service-ca.crt\""
Jul 16 22:45:18.316804 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.316789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xsq47\"/\"default-dockercfg-8flnx\""
Jul 16 22:45:18.328699 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.328679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"]
Jul 16 22:45:18.343288 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.343262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c" event={"ID":"a95f3b78-597b-43f7-b4d2-6276f19afefa","Type":"ContainerDied","Data":"7187c0cb84fa9dc312ecf4dc2c15ee4bbeca32226b3e685722e60c8b479b61ef"}
Jul 16 22:45:18.343370 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.343288    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"
Jul 16 22:45:18.343370 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.343326    2566 scope.go:117] "RemoveContainer" containerID="4555a1f903e3589d74f45fc18fa7fe040a5ffb35e3f3c9ea230204905af3b8de"
Jul 16 22:45:18.372871 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.372846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-lib-modules\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.372985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.372888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svckc\" (UniqueName: \"kubernetes.io/projected/ad23ffef-4c19-48f0-9371-6f4feda1b28d-kube-api-access-svckc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.372985 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.372975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-podres\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.373077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.373044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-sys\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.373077 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.373070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-proc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.398950 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.398928    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:45:18.404086 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.404063    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eea3cfe3-7l46c"]
Jul 16 22:45:18.474312 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-sys\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474418 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-proc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474418 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-lib-modules\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-sys\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svckc\" (UniqueName: \"kubernetes.io/projected/ad23ffef-4c19-48f0-9371-6f4feda1b28d-kube-api-access-svckc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-lib-modules\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-proc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474568 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-podres\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.474792 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.474623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad23ffef-4c19-48f0-9371-6f4feda1b28d-podres\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.483440 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.483420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svckc\" (UniqueName: \"kubernetes.io/projected/ad23ffef-4c19-48f0-9371-6f4feda1b28d-kube-api-access-svckc\") pod \"perf-node-gather-daemonset-nrz6z\" (UID: \"ad23ffef-4c19-48f0-9371-6f4feda1b28d\") " pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.621329 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.621270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:18.772902 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:18.772877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"]
Jul 16 22:45:18.774426 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:45:18.774400    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podad23ffef_4c19_48f0_9371_6f4feda1b28d.slice/crio-d04922c8b4a61e45198a9f5bda0cd54385fa4317dccd62351cdbb0b7c3922cb4 WatchSource:0}: Error finding container d04922c8b4a61e45198a9f5bda0cd54385fa4317dccd62351cdbb0b7c3922cb4: Status 404 returned error can't find the container with id d04922c8b4a61e45198a9f5bda0cd54385fa4317dccd62351cdbb0b7c3922cb4
Jul 16 22:45:19.350407 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.350367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z" event={"ID":"ad23ffef-4c19-48f0-9371-6f4feda1b28d","Type":"ContainerStarted","Data":"3ced62352ab2587ec0623f7fde967c1db89983606f6b98eaa60875cbae6f67f9"}
Jul 16 22:45:19.350613 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.350425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z" event={"ID":"ad23ffef-4c19-48f0-9371-6f4feda1b28d","Type":"ContainerStarted","Data":"d04922c8b4a61e45198a9f5bda0cd54385fa4317dccd62351cdbb0b7c3922cb4"}
Jul 16 22:45:19.350613 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.350573    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:19.370110 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.370057    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z" podStartSLOduration=1.3700437810000001 podStartE2EDuration="1.370043781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:19.367647149 +0000 UTC m=+3477.877514230" watchObservedRunningTime="2026-07-16 22:45:19.370043781 +0000 UTC m=+3477.879910862"
Jul 16 22:45:19.824233 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.824200    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/1.log"
Jul 16 22:45:19.831444 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:19.831422    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-swksv_50807f4a-c8ab-411d-9519-fb3f9cc88f30/console-operator/2.log"
Jul 16 22:45:20.011218 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:20.011180    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" path="/var/lib/kubelet/pods/a95f3b78-597b-43f7-b4d2-6276f19afefa/volumes"
Jul 16 22:45:20.308508 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:20.308476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7755fb5649-bjhcn_b51aa90f-f006-4eb8-aea6-a2a8f5afb138/console/0.log"
Jul 16 22:45:21.015488 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:21.015371    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-zmcsq_c2fa3599-1965-4a8d-965d-62a32be2d750/volume-data-source-validator/0.log"
Jul 16 22:45:22.298260 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:22.298230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vdlb_8b7bf600-8418-4011-85ea-de76f6ee0ea3/dns/0.log"
Jul 16 22:45:22.343915 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:22.343884    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vdlb_8b7bf600-8418-4011-85ea-de76f6ee0ea3/kube-rbac-proxy/0.log"
Jul 16 22:45:22.488564 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:22.488537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjf7s_e901dacd-baf6-481d-946f-1609920866a0/dns-node-resolver/0.log"
Jul 16 22:45:22.573208 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:22.573136    2566 scope.go:117] "RemoveContainer" containerID="f257b016562bb3fab292c5be996e00716f139cfb1dd0b022472f53610cabd589"
Jul 16 22:45:23.173556 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:23.173523    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxjdn_6d6ed8a8-5f2f-4b83-9409-c7ffc432f1ea/node-ca/0.log"
Jul 16 22:45:24.137633 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.137601    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47ptlz_dc34873d-40f2-4153-a382-d69f6c703405/istio-proxy/0.log"
Jul 16 22:45:24.392216 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.392135    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-2tn7p_d303fb66-62db-49a0-afce-89f3313d9ec6/kube-auth-proxy/0.log"
Jul 16 22:45:24.467518 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.467487    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/0.log"
Jul 16 22:45:24.523342 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.523306    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/1.log"
Jul 16 22:45:24.571045 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.571002    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-6jvh5_b65f15b0-318e-41ba-8fbd-6bb7d008343d/payload-pre-processing/0.log"
Jul 16 22:45:24.612399 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.612375    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6s658_5c2dae41-16b6-463c-a7b1-8ce8579a0c60/payload-processing/0.log"
Jul 16 22:45:24.641897 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:24.641874    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddb94bc54-9kx9k_e4b0774f-37d5-498e-a1ff-69f5189b5398/router/0.log"
Jul 16 22:45:25.365764 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.365734    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xsq47/perf-node-gather-daemonset-nrz6z"
Jul 16 22:45:25.406834 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.406790    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jkp4f_2acd24b3-f582-4735-a4eb-96c86a6472cf/serve-healthcheck-canary/0.log"
Jul 16 22:45:25.942904 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.942862    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8bbmz_d3978b80-d899-4fa8-86b0-48885a155b24/insights-operator/0.log"
Jul 16 22:45:25.943992 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.943961    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8bbmz_d3978b80-d899-4fa8-86b0-48885a155b24/insights-operator/1.log"
Jul 16 22:45:25.964220 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.964199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6849m_6c1ef0ce-398e-4a2a-99f0-071806905ba4/kube-rbac-proxy/0.log"
Jul 16 22:45:25.988326 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:25.988303    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6849m_6c1ef0ce-398e-4a2a-99f0-071806905ba4/exporter/0.log"
Jul 16 22:45:26.012086 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.012065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6849m_6c1ef0ce-398e-4a2a-99f0-071806905ba4/extractor/0.log"
Jul 16 22:45:26.832908 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.832874    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"]
Jul 16 22:45:26.833337 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.833322    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:26.833337 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.833339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:26.833438 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.833429    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95f3b78-597b-43f7-b4d2-6276f19afefa" containerName="revoke-keys"
Jul 16 22:45:26.838295 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.838279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"
Jul 16 22:45:26.841351 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.841333    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vr6jw\""
Jul 16 22:45:26.847084 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.847060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"]
Jul 16 22:45:26.956689 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:26.956660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb237997-c919-414c-98e4-f823f53a3248-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd\" (UID: \"eb237997-c919-414c-98e4-f823f53a3248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"
Jul 16 22:45:27.057420 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:27.057385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb237997-c919-414c-98e4-f823f53a3248-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd\" (UID: \"eb237997-c919-414c-98e4-f823f53a3248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"
Jul 16 22:45:27.058069 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:27.058049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb237997-c919-414c-98e4-f823f53a3248-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd\" (UID: \"eb237997-c919-414c-98e4-f823f53a3248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"
Jul 16 22:45:27.149768 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:27.149678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"
Jul 16 22:45:27.504175 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:27.504151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd"]
Jul 16 22:45:27.505811 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:45:27.505786    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb237997_c919_414c_98e4_f823f53a3248.slice/crio-ae6fb62d36cbbe76e27464f3ca04458dfa2777d58d97d495bffdf18f4011c623 WatchSource:0}: Error finding container ae6fb62d36cbbe76e27464f3ca04458dfa2777d58d97d495bffdf18f4011c623: Status 404 returned error can't find the container with id ae6fb62d36cbbe76e27464f3ca04458dfa2777d58d97d495bffdf18f4011c623
Jul 16 22:45:28.406849 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.406759    2566 generic.go:356] "Generic (PLEG): container finished" podID="eb237997-c919-414c-98e4-f823f53a3248" containerID="4dcd077fd90886085e7c633f9756feab4cf2074d7a755e4e9a206f390ded36a5" exitCode=6
Jul 16 22:45:28.407357 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.407273    2566 scope.go:117] "RemoveContainer" containerID="4dcd077fd90886085e7c633f9756feab4cf2074d7a755e4e9a206f390ded36a5"
Jul 16 22:45:28.407542 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.406895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd" event={"ID":"eb237997-c919-414c-98e4-f823f53a3248","Type":"ContainerDied","Data":"4dcd077fd90886085e7c633f9756feab4cf2074d7a755e4e9a206f390ded36a5"}
Jul 16 22:45:28.407633 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.407560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd" event={"ID":"eb237997-c919-414c-98e4-f823f53a3248","Type":"ContainerStarted","Data":"ae6fb62d36cbbe76e27464f3ca04458dfa2777d58d97d495bffdf18f4011c623"}
Jul 16 22:45:28.778181 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.778153    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w5c8b_f938169b-f9fb-4385-a27c-4e7450ed1366/manager/0.log"
Jul 16 22:45:28.957622 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:28.957590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47ptlz_dc34873d-40f2-4153-a382-d69f6c703405/istio-proxy/0.log"
Jul 16 22:45:29.005408 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.005355    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-74qf2_bb8353bb-5367-4dc1-8d85-3cead16b9d5b/manager/0.log"
Jul 16 22:45:29.173394 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.173367    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-2tn7p_d303fb66-62db-49a0-afce-89f3313d9ec6/kube-auth-proxy/0.log"
Jul 16 22:45:29.176864 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.176843    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58f7bcf9f5-sxhf9_b555e970-cac6-4667-8742-523d8be45e71/manager/0.log"
Jul 16 22:45:29.207761 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.207739    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/0.log"
Jul 16 22:45:29.233801 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.233771    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"]
Jul 16 22:45:29.238401 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.238373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"
Jul 16 22:45:29.246551 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.246450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"]
Jul 16 22:45:29.295279 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.295254    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/1.log"
Jul 16 22:45:29.335393 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.335366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-6jvh5_b65f15b0-318e-41ba-8fbd-6bb7d008343d/payload-pre-processing/0.log"
Jul 16 22:45:29.352338 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.352302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nlx76_75ce86e8-4a0f-4bed-99cc-8a206c2216cf/manager/0.log"
Jul 16 22:45:29.366637 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.366614    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6s658_5c2dae41-16b6-463c-a7b1-8ce8579a0c60/payload-processing/0.log"
Jul 16 22:45:29.376944 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.376924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811a47ef-d45b-4e18-8fd8-50c668727dc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-5656p\" (UID: \"811a47ef-d45b-4e18-8fd8-50c668727dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"
Jul 16 22:45:29.384219 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.384198    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-5jrpx_40c0974e-151e-40e5-a6c3-bad33f4c32b0/manager/0.log"
Jul 16 22:45:29.384623 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.384605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddb94bc54-9kx9k_e4b0774f-37d5-498e-a1ff-69f5189b5398/router/0.log"
Jul 16 22:45:29.412846 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.412831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-574666dc8f-5jrpx_40c0974e-151e-40e5-a6c3-bad33f4c32b0/copy-manifests/0.log"
Jul 16 22:45:29.414914 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.414893    2566 generic.go:356] "Generic (PLEG): container finished" podID="eb237997-c919-414c-98e4-f823f53a3248" containerID="167a23b0030a2ea5f3004a97bc356d5186bdbb69c7039b147e1056cdc9297883" exitCode=6
Jul 16 22:45:29.415022 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.414932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd" event={"ID":"eb237997-c919-414c-98e4-f823f53a3248","Type":"ContainerDied","Data":"167a23b0030a2ea5f3004a97bc356d5186bdbb69c7039b147e1056cdc9297883"}
Jul 16 22:45:29.415022 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.414993    2566 scope.go:117] "RemoveContainer" containerID="4dcd077fd90886085e7c633f9756feab4cf2074d7a755e4e9a206f390ded36a5"
Jul 16 22:45:29.415152 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.415136    2566 scope.go:117] "RemoveContainer" containerID="167a23b0030a2ea5f3004a97bc356d5186bdbb69c7039b147e1056cdc9297883"
Jul 16 22:45:29.415406 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:29.415385    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd_odh-ai-gateway-infra(eb237997-c919-414c-98e4-f823f53a3248)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd" podUID="eb237997-c919-414c-98e4-f823f53a3248"
Jul 16 22:45:29.478333 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.478300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811a47ef-d45b-4e18-8fd8-50c668727dc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-5656p\" (UID: \"811a47ef-d45b-4e18-8fd8-50c668727dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"
Jul 16 22:45:29.478938 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.478919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/811a47ef-d45b-4e18-8fd8-50c668727dc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67907cc2-5656p\" (UID: \"811a47ef-d45b-4e18-8fd8-50c668727dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"
Jul 16 22:45:29.554628 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.554564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"
Jul 16 22:45:29.695623 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:29.695599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p"]
Jul 16 22:45:29.697350 ip-10-0-140-81 kubenswrapper[2566]: W0716 22:45:29.697320    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811a47ef_d45b_4e18_8fd8_50c668727dc5.slice/crio-9b6872513b7bd63acda3830efdf4950f9a96433216b447107a8ba0e329e6e5ee WatchSource:0}: Error finding container 9b6872513b7bd63acda3830efdf4950f9a96433216b447107a8ba0e329e6e5ee: Status 404 returned error can't find the container with id 9b6872513b7bd63acda3830efdf4950f9a96433216b447107a8ba0e329e6e5ee
Jul 16 22:45:30.422075 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.422052    2566 scope.go:117] "RemoveContainer" containerID="167a23b0030a2ea5f3004a97bc356d5186bdbb69c7039b147e1056cdc9297883"
Jul 16 22:45:30.422439 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:30.422283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd_odh-ai-gateway-infra(eb237997-c919-414c-98e4-f823f53a3248)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f2f32f5-v2gcd" podUID="eb237997-c919-414c-98e4-f823f53a3248"
Jul 16 22:45:30.423379 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.423360    2566 generic.go:356] "Generic (PLEG): container finished" podID="811a47ef-d45b-4e18-8fd8-50c668727dc5" containerID="694daab9f5cd1b7b6cd6f8a0a2d5e8002c61910a28e9806e9c93f75a5483231c" exitCode=6
Jul 16 22:45:30.423485 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.423388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p" event={"ID":"811a47ef-d45b-4e18-8fd8-50c668727dc5","Type":"ContainerDied","Data":"694daab9f5cd1b7b6cd6f8a0a2d5e8002c61910a28e9806e9c93f75a5483231c"}
Jul 16 22:45:30.423485 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.423425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p" event={"ID":"811a47ef-d45b-4e18-8fd8-50c668727dc5","Type":"ContainerStarted","Data":"9b6872513b7bd63acda3830efdf4950f9a96433216b447107a8ba0e329e6e5ee"}
Jul 16 22:45:30.423610 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.423594    2566 scope.go:117] "RemoveContainer" containerID="694daab9f5cd1b7b6cd6f8a0a2d5e8002c61910a28e9806e9c93f75a5483231c"
Jul 16 22:45:30.613186 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.613098    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf47ptlz_dc34873d-40f2-4153-a382-d69f6c703405/istio-proxy/0.log"
Jul 16 22:45:30.877360 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.877290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f57477db-2tn7p_d303fb66-62db-49a0-afce-89f3313d9ec6/kube-auth-proxy/0.log"
Jul 16 22:45:30.947571 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:30.947540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/0.log"
Jul 16 22:45:31.005786 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.005758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzwj_456b15ec-9505-47ae-97a2-ec32e7c811d0/istio-proxy/1.log"
Jul 16 22:45:31.049006 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.048973    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-6jvh5_b65f15b0-318e-41ba-8fbd-6bb7d008343d/payload-pre-processing/0.log"
Jul 16 22:45:31.056817 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.056792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dm7dr_84c5d4b0-bbb7-4326-9d00-1f3a7bfb3149/openshift-lws-operator/0.log"
Jul 16 22:45:31.101530 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.101505    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6s658_5c2dae41-16b6-463c-a7b1-8ce8579a0c60/payload-processing/0.log"
Jul 16 22:45:31.129957 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.129848    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6ddb94bc54-9kx9k_e4b0774f-37d5-498e-a1ff-69f5189b5398/router/0.log"
Jul 16 22:45:31.434991 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.434915    2566 generic.go:356] "Generic (PLEG): container finished" podID="811a47ef-d45b-4e18-8fd8-50c668727dc5" containerID="9b002118803a46fe59a70ddfb8738005aac167b3ca7ce01ade49510cc8efd7ae" exitCode=6
Jul 16 22:45:31.435389 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.434999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p" event={"ID":"811a47ef-d45b-4e18-8fd8-50c668727dc5","Type":"ContainerDied","Data":"9b002118803a46fe59a70ddfb8738005aac167b3ca7ce01ade49510cc8efd7ae"}
Jul 16 22:45:31.435389 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.435058    2566 scope.go:117] "RemoveContainer" containerID="694daab9f5cd1b7b6cd6f8a0a2d5e8002c61910a28e9806e9c93f75a5483231c"
Jul 16 22:45:31.435389 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:31.435258    2566 scope.go:117] "RemoveContainer" containerID="9b002118803a46fe59a70ddfb8738005aac167b3ca7ce01ade49510cc8efd7ae"
Jul 16 22:45:31.435583 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:31.435524    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-5656p_odh-ai-gateway-infra(811a47ef-d45b-4e18-8fd8-50c668727dc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p" podUID="811a47ef-d45b-4e18-8fd8-50c668727dc5"
Jul 16 22:45:32.442188 ip-10-0-140-81 kubenswrapper[2566]: I0716 22:45:32.442162    2566 scope.go:117] "RemoveContainer" containerID="9b002118803a46fe59a70ddfb8738005aac167b3ca7ce01ade49510cc8efd7ae"
Jul 16 22:45:32.442601 ip-10-0-140-81 kubenswrapper[2566]: E0716 22:45:32.442381    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-67907cc2-5656p_odh-ai-gateway-infra(811a47ef-d45b-4e18-8fd8-50c668727dc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67907cc2-5656p" podUID="811a47ef-d45b-4e18-8fd8-50c668727dc5"